Haptic perception is highly important for immersive teleoperation of robots, especially for accomplishing manipulation tasks. We propose a low-cost haptic sensing and rendering system, which is capable of detecting and displaying surface roughness. As the robot fingertip moves across a surface of interest, two microphones capture sound coupled directly through the fingertip and through the air, respectively. A learning-based detector system analyzes the data in real time and gives roughness estimates with both high temporal resolution and low latency. Finally, an audio-based vibrational actuator displays the result to the human operator. We demonstrate the effectiveness of our system through lab experiments and our winning entry in the ANA Avatar XPRIZE competition finals, where briefly trained judges solved a roughness-based selection task even without additional vision feedback. We publish our dataset used for training and evaluation together with our trained models to enable reproducibility of results.
University of Bristol（布里斯托大学）；Ultraleap Ltd.（Ultraleap有限公司）
Mid-air haptics create a new mode of feedback to allow people to feel tactile sensations in the air. Ultrasonic arrays focus acoustic radiation pressure in space, to induce tactile sensation from the resulting skin deflection. In this work, we present a low-cost tactile robot to test mid-air haptics. By combining a desktop robot arm with a 3D-printed biomimetic tactile sensor, we developed a system that can sense, map, and visualize mid-air haptic sensations created by an ultrasonic transducer array. We evaluate our tactile robot by testing it on a variety of mid-air haptic stimuli, including …
Wrist-worn haptic interfaces can deliver a wide range of tactile cues for communication of information and interaction with virtual objects. Unlike fingertips, the wrist and forearm provide a considerably large area of skin that allows the placement of multiple haptic actuators as a display for enriching tactile information transfer with minimal encumbrance. Existing multi-degree-of-freedom (DoF) wrist-worn devices employ traditional rigid robotic mechanisms and electric motors that limit their versatility, miniaturization, distribution, and assembly. Alternative solutions based on soft elasto…
Mobile telepresence robots (MTRs) have become increasingly popular in the expanding world of remote work, providing new avenues for people to actively participate in activities at a distance. However, humans operating MTRs often have difficulty navigating in densely populated environments due to limited situation awareness and narrow field-of-view, which reduces user acceptance and satisfaction. Shared autonomy in navigation has been studied primarily in static environments or in situations where only one pedestrian interacts with the robot. We present a multimodal shared autonomy approach, le…
Koc University（科奇大学）；University of Lille（里尔大学）；École de Technologie Supérieure（高等技术学院）；Pohang University of Science and Technology (POSTECH)（浦项科技大学（POSTECH））
We review the current technology underlying surface haptics that converts passive touch surfaces to active ones (machine haptics), our perception of tactile stimuli displayed through active touch surfaces (human haptics), their potential applications (human-machine interaction), and finally the challenges ahead of us in making them available through commercial systems. This review primarily covers the tactile interactions of human fingers or hands with surface-haptics displays by focusing on the three most popular actuation methods: vibrotactile, electrostatic, and ultrasonic.
University of Liverpool（利物浦大学）；King’s College London（伦敦国王学院）；University of Leeds（利兹大学）；Xi’an Jiaotong University（西安交通大学）
To assist robots in teleoperation tasks, haptic rendering which allows human operators access a virtual touch feeling has been developed in recent years. Most previous haptic rendering methods strongly rely on data collected by tactile sensors. However, tactile data is not widely available for robots due to their limited reachable space and the restrictions of tactile sensors. To eliminate the need for tactile data, in this paper we propose a novel method named as Vis2Hap to generate haptic rendering from visual inputs that can be obtained from a distance without physical interaction. We take …
Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation
Queen Mary University of London（伦敦玛丽女王大学）；School of Engineering and Materials Science（工程与材料科学学院）；Humanoid AI（人形人工智能公司）；King’s College London（伦敦国王学院）；School of Biomedical Engineering and Imaging Sciences（生物医学工程与影像科学学院）
Integrating and Evaluating Visuo-tactile Sensing with Haptic Feedback for Teleoperated Robot Manipulation
Technische Universität Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Hessian Centre for Artificial Intelligence (Hessian.AI)（黑森州人工智能中心）；Centre for Cognitive Science (CogSci)（认知科学中心）；Nagoya University（名古屋大学）
Stanford University（斯坦福大学）；University of Southern California（南加州大学）
A common and effective form of social touch is stroking on the forearm. We seek to replicate this stroking sensation using haptic illusions. This work compares two methods that provide sequential discrete stimulation: sequential normal indentation and sequential lateral skin-slip using discrete actuators. Our goals are to understand which form of stimulation more effectively creates a continuous stroking sensation, and how many discrete contact points are needed. We performed a study with 20 participants in which they rated sensations from the haptic devices on continuity and pleasantness. We …
Nagoya University（名古屋大学）；Technical University of Darmstadt（达姆施塔特工业大学）
Vision-based tactile sensors have gained extensive attention in the robotics community. The sensors are highly expected to be capable of extracting contact information i.e. haptic information during in-hand manipulation. This nature of tactile sensors makes them a perfect match for haptic feedback applications. In this paper, we propose a contact force estimation method using the vision-based tactile sensor DIGIT, and apply it to a position-force teleoperation architecture for force feedback. The force estimation is done by building a depth map for DIGIT gel surface deformation measurement and…
University of California at Berkeley（加州大学伯克利分校）；University of Pennsylvania（宾夕法尼亚大学）
Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces. Additionally, for certain tasks, robots may need to know the haptic properties of an object before touching it. To enable better tactile understanding for robots, we propose a method of classifying surfaces with haptic adjectives (e.g., compressible or smooth) from both visual and physical interaction data. Humans typically combine visual predictions and feedback from physical interactions to accurately predict haptic properties and interact with the world. Inspired by …
A Haptic Robot Finger Designed for Guqin Instrument Playing
The Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）；The Chinese University of Hong Kong - Shenzhen（香港中文大学（深圳））；School of Physics and Optoelectronic Engineering, Shenzhen University（深圳大学物理与光电工程学院）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Haptic exploration is a key skill for both robots and humans to discriminate and handle unknown objects or to recognize familiar objects. Its active nature is evident in humans who from early on reliably acquire sophisticated sensory-motor capabilities for active exploratory touch and directed manual exploration that associates surfaces and object properties with their spatial locations. This is in stark contrast to robotics. In this field, the relative lack of good real-world interaction models - along with very restricted sensors and a scarcity of suitable training data to leverage machine l…
LIRMM, University of Montpellier, CNRS（蒙彼利埃大学法国国家科学研究中心LIRMM实验室）；Stanford University（斯坦福大学）
The current generation of surgeons requires extensive training in teleoperation to develop specific dexterous skills, which are independent of medical knowledge. Training curricula progress from manipulation tasks to simulated surgical tasks but are limited in time. To tackle this, we propose to integrate surgical robotic training together with Haptic Feedback (HF) to improve skill acquisition. This paper present the initial but promising results of our haptic device designed to support in the training of surgical gestures. Our ongoing work is related to integrate the HF in the RAVEN II platfo…
Teleoperation of robotic systems for precise and delicate object grasping requires high-fidelity haptic feedback to obtain comprehensive real-time information about the grasp. In such cases, the most common approach is to use kinesthetic feedback. However, a single contact point information is insufficient to detect the dynamically changing shape of soft objects. This paper proposes a novel telemanipulation system that provides kinesthetic and cutaneous stimuli to the user's hand to achieve accurate liquid dispensing by dexterously manipulating the deformable object (i.e., pipette). The experi…
In this paper we propose a novel method for in-hand object recognition. The method is composed of a grasp stabilization controller and two exploratory behaviours to capture the shape and the softness of an object. Grasp stabilization plays an important role in recognizing objects. First, it prevents the object from slipping and facilitates the exploration of the object. Second, reaching a stable and repeatable position adds robustness to the learning algorithm and increases invariance with respect to the way in which the robot grasps the object. The stable poses are estimated using a Gaussian …
A Biomimetic Fingerprint for Robotic Tactile Sensing
Faculty of Mechatronic \& Micro-Mechatronic Systems Hochschule Karlsruhe (HKA) Karlsruhe, Germany；Deutsches Forschungszentrum für Künstliche Intelligenz GmbH；L3S Research Center, Leibniz Universität Hannover Hannover, Germany
Space Center of Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术研究院空间中心）；Center for Design, Manufacturing and Materials（设计、制造与材料中心）
Modern industry still relies on manual manufacturing operations and safe human-robot interaction is of great interest nowadays. Speed and Separation Monitoring (SSM) allows close and efficient collaborative scenarios by maintaining a protective separation distance during robot operation. The paper focuses on a novel approach to strengthen the SSM safety requirements by introducing haptic feedback to a robotic cell worker. Tactile stimuli provide early warning of dangerous movements and proximity to the robot, based on the human reaction time and instantaneous velocities of robot and operator. …
Skolkovo Institute of Science and Technology (Skoltech)（斯科尔科沃科学技术研究院（Skoltech））
TouchVR is a novel wearable haptic interface which can deliver multimodal tactile stimuli on the palm by DeltaTouch haptic display and vibrotactile feedback on the fingertips by vibration motors for the Virtual Reality (VR) user. DeltaTouch display is capable of generating 3D force vector at the contact point and presenting multimodal tactile sensation of weight, slippage, encounter, softness, and texture. The VR system consists of HTC Vive Pro base stations and head-mounted display (HMD), and Leap Motion controller for tracking the user's hands motion in VR. The MatrixTouch, BallFeel, and Rob…
Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）
The lack of haptically aware upper-limb prostheses forces amputees to rely largely on visual cues to complete activities of daily living. In contrast, able-bodied individuals inherently rely on conscious haptic perception and automatic tactile reflexes to govern volitional actions in situations that do not allow for constant visual attention. We therefore propose a myoelectric prosthesis system that reflects these concepts to aid manipulation performance without direct vision. To implement this design, we built two fabric-based tactile sensors that measure contact location along the palmar and…
Technical University of Munich（慕尼黑工业大学）；ABB Corporate Research（ABB集团研究院）；University of Campania Luigi Vanvitelli（坎帕尼亚大学路易吉·万维泰利）；Institute of Robotics and Mechatronics, German Aerospace Center（德国航空航天中心机器人与机电一体化研究所）
In this work, we introduce the problem of cross-modal visuo-tactile object recognition with robotic active exploration. With this term, we mean that the robot observes a set of objects with visual perception and, later on, it is able to recognize such objects only with tactile exploration, without having touched any object before. Using a machine learning terminology, in our application we have a visual training set and a tactile test set, or vice versa. To tackle this problem, we propose an approach constituted by four steps: finding a visuo-tactile common representation, defining a suitable …
California Institute of Technology（加州理工学院）；University of Connecticut（康涅狄格大学）；Facebook Inc.（脸书公司）
Haptic feedback is the most significant sensory interface following visual cues. Developing thin, flexible surfaces that function as haptic interfaces is important for augmenting virtual reality, wearable devices, robotics and prostheses. For example, adding a haptic feedback interface to prosthesis could improve their acceptance among amputees. State of the art programmable interfaces targeting the skin feel-of-touch through mechano-receptors are limited by inadequate sensory feedback, cumbersome mechanisms or narrow frequency of operation. Here, we present a flexible metasurface as a generic…
Skolkovo Institute of Science and Technology（斯科尔科沃科技学院）
We propose a novel concept of augmented reality (AR) human-drone interaction driven by RL-based swarm behavior to achieve intuitive and immersive control of a swarm formation of unmanned aerial vehicles. The DroneARchery system developed by us allows the user to quickly deploy a swarm of drones, generating flight paths simulating archery. The haptic interface LinkGlide delivers a tactile stimulus of the bowstring tension to the forearm to increase the precision of aiming. The swarm of released drones dynamically avoids collisions between each other, the drone following the user, and external o…
Towards Tangible Immersion for Cobot Programming-by-Demonstration: Visual, Tactile and Haptic Interfaces for Mixed-Reality Cobot Automation in Semiconductor Manufacturing
Intel Labs, Intelligent System Research, Human-Robot Collaboration（英特尔实验室、智能系统研究、人机协作）
Tactile interaction plays a crucial role in interactions between people. Touch can, for example, help people calm down and lower physiological stress responses. Consequently, it is believed that tactile and haptic interaction matter also in human-robot interaction. We study if the intensity of the tactile interaction has an impact on people, and do so by studying whether different intensities of tactile interaction modulate physiological measures and task performance. We use a paradigm in which a small humanoid robot is used to encourage risk-taking behaviour, relying on peer encouragement to …
King’s College London（伦敦国王学院）；University of Liverpool（利物浦大学）；Queen Mary University of London（伦敦玛丽女王大学）
For humans, both the proprioception and touch sensing are highly utilized when performing haptic perception. However, most approaches in robotics use only either proprioceptive data or touch data in haptic object recognition. In this paper, we present a novel method named Iterative Closest Labeled Point (iCLAP) to link the kinesthetic cues and tactile patterns fundamentally and also introduce its extensions to recognize object shapes. In the training phase, the iCLAP first clusters the features of tactile readings into a codebook and assigns these features with distinct label numbers. A 4D poi…
3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception
State Key Laboratory of Digital Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology（数字制造装备与技术国家重点实验室，机械科学与工程学院，华中科技大学）
A wearable haptic device for edge and surface simulation
Institute of Mechanical Intelligence, School of Advanced Studies Sant'Anna (SSSA)（圣安娜高等研究学院机械智能研究所）；CAS Key Laboratory of Mechanical Behavior and Design of Materials, Institute of Humanoid Robots, School of Engineering Sciences, University of Science and Technology of China（中国科学院材料力学行为和设计重点实验室，中国科学技术大学工程科学学院仿生机器人研究所）
German Aerospace Center (DLR)（德国航空航天中心（DLR））；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT））；University of Twente (UT)（特文特大学（UT））；Technical University of Munich (TUM)（慕尼黑工业大学（TUM））
This article presents a novel telepresence system for advancing aerial manipulation in dynamic and unstructured environments. The proposed system not only features a haptic device, but also a virtual reality (VR) interface that provides real-time 3D displays of the robot's workspace as well as a haptic guidance to its remotely located operator. To realize this, multiple sensors namely a LiDAR, cameras and IMUs are utilized. For processing of the acquired sensory data, pose estimation pipelines are devised for industrial objects of both known and unknown geometries. We further propose an active…
Curriculum Is More Influential Than Haptic Information During Reinforcement Learning of Object Manipulation Against Gravity
University of Southern California（南加州大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of California Santa Cruz（加州大学圣克鲁兹分校）；University of Cincinnati（辛辛那提大学）
Multimodal object recognition is still an emerging field. Thus, publicly available datasets are still rare and of small size. This dataset was developed to help fill this void and presents multimodal data for 63 objects with some visual and haptic ambiguity. The dataset contains visual, kinesthetic and tactile (audio/vibrations) data. To completely solve sensory ambiguity, sensory integration/fusion would be required. This report describes the creation and structure of the dataset. The first section explains the underlying approach used to capture the visual and haptic properties of the object…
