Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing
Institute for Dynamic Systems and Control, ETH Zurich（苏黎世联邦理工学院动态系统与控制研究所）
Carmelo Sferrazza, Thomas Bi, Raffaello D'Andrea
备注：
This work has been submitted to the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) for possible publication. Accompanying video: https://youtu.be/dDTga9PgWS0
Data-driven approaches to tactile sensing aim to overcome the complexity of accurately modeling contact with soft materials. However, their widespread adoption is impaired by concerns about data efficiency and the capability to generalize when applied to various tasks. This paper focuses on both these aspects with regard to a vision-based tactile sensor, which aims to reconstruct the distribution of the three-dimensional contact forces applied on its soft surface. Accurate models for the soft materials and the camera projection, derived via state-of-the-art techniques in the respective domains, are employed to generate a dataset in simulation. A strategy is proposed to train a tailored deep neural network entirely from the simulation data. The resulting learning architecture is directly transferable across multiple tactile sensors without further training and yields accurate predictions on real data, while showing promising generalization capabilities to unseen contact conditions.
University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
High-resolution optical tactile sensors are increasingly used in robotic learning environments due to their ability to capture large amounts of data directly relating to agent-environment interaction. However, there is a high barrier of entry to research in this area due to the high cost of tactile robot platforms, specialised simulation software, and sim-to-real methods that lack generality across different sensors. In this letter we extend the Tactile Gym simulator to include three new optical tactile sensors (TacTip, DIGIT and DigiTac) of the two most popular types, Gelsight-style (image-sh…
Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）
The integration of visual-tactile stimulus is common while humans performing daily tasks. In contrast, using unimodal visual or tactile perception limits the perceivable dimensionality of a subject. However, it remains a challenge to integrate the visual and tactile perception to facilitate robotic tasks. In this paper, we propose a novel framework for the cross-modal sensory data generation for visual and tactile perception. Taking texture perception as an example, we apply conditional generative adversarial networks to generate pseudo visual images or tactile outputs from data of the other m…
TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳研究院）；Meituan Academy of Robotics（美团机器人研究院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
This report presents the experimental methodology and a step-by-step guide for gathering data on how aging influences tactile surface perception in decision and action. The experiments consist of a set of trials in which the ability to distinguish tactile stimuli is investigated. A robot arm is used to provide a systematic and unbiased presentation of the stimuli.
In this paper, we present an approach to tactile pose estimation from the first touch for known objects. First, we create an object-agnostic map from real tactile observations to contact shapes. Next, for a new object with known geometry, we learn a tailored perception model completely in simulation. To do so, we simulate the contact shapes that a dense set of object poses would produce on the sensor. Then, given a new contact shape obtained from the sensor output, we match it against the pre-computed set using the object-specific embedding learned purely in simulation using contrastive learni…
University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
This article describes a new way of controlling robots using soft tactile sensors: pose-based tactile servo (PBTS) control. The basic idea is to embed a tactile perception model for estimating the sensor pose within a servo control loop that is applied to local object features such as edges and surfaces. PBTS control is implemented with a soft curved optical tactile sensor (the BRL TacTip) using a convolutional neural network trained to be insensitive to shear. In consequence, robust and accurate controlled motion over various complex 3D objects is attained. First, we review tactile servoing a…
Facebook AI Research（Facebook人工智能研究院）；McGill University（麦吉尔大学）；University of California, Berkeley（加州大学伯克利分校）
Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. However, in 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings, leaving the active exploration of the shape largely unexplored. Inactive touch sensing for 3D reconstruction, the goal is to actively select the tactile readings that maximize the improvement in shape reconstruction accuracy. However, the development of deep learning-based active touch models is largely limit…
Koc University（科奇大学）；University of Lille（里尔大学）；École de Technologie Supérieure（高等技术学院）；Pohang University of Science and Technology (POSTECH)（浦项科技大学（POSTECH））
We review the current technology underlying surface haptics that converts passive touch surfaces to active ones (machine haptics), our perception of tactile stimuli displayed through active touch surfaces (human haptics), their potential applications (human-machine interaction), and finally the challenges ahead of us in making them available through commercial systems. This review primarily covers the tactile interactions of human fingers or hands with surface-haptics displays by focusing on the three most popular actuation methods: vibrotactile, electrostatic, and ultrasonic.
Let Robots Feel Your Touch: Visuo-Tactile Cortical Alignment for Embodied Mirror Resonance
Institute for AI Industry Research, Tsinghua University（清华大学人工智能产业研究院）；Key Laboratory of Biomedical Engineering of Hainan Province, School of Biomedical Engineering, Hainan University（海南省生物医学工程重点实验室，海南大学生物医学工程学院）；School of New Media Art and Design, Beihang University（北航艺术与设计学院）；MoE Key Laboratory for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology（教育部生物医学光子学重点实验室，武汉光电研究所，华中科技大学）
Humans perceive the world using multi-modal sensory inputs such as vision, audition, and touch. In this work, we investigate the cross-modal connection between vision and touch. The main challenge in this cross-domain modeling task lies in the significant scale discrepancy between the two: while our eyes perceive an entire visual scene at once, humans can only feel a small region of an object at any given moment. To connect vision and touch, we introduce new tasks of synthesizing plausible tactile signals from visual inputs as well as imagining how we interact with objects given tactile data a…
University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
Deep learning combined with high-resolution tactile sensing could lead to highly capable dexterous robots. However, progress is slow because of the specialist equipment and expertise. The DIGIT tactile sensor offers low-cost entry to high-resolution touch using GelSight-type sensors. Here we customize the DIGIT to have a 3D-printed sensing surface based on the TacTip family of soft biomimetic optical tactile sensors. The DIGIT-TacTip (DigiTac) enables direct comparison between these distinct tactile sensor types. For this comparison, we introduce a tactile robot system comprising a desktop arm…
University of California, Berkeley（加利福尼亚大学伯克利分校）；Facebook AI Research（Facebook人工智能研究院）
Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dyn…
We present MidasTouch, a tactile perception system for online global localization of a vision-based touch sensor sliding on an object surface. This framework takes in posed tactile images over time, and outputs an evolving distribution of sensor pose on the object's surface, without the need for visual priors. Our key insight is to estimate local surface geometry with tactile sensing, learn a compact representation for it, and disambiguate these signals over a long time horizon. The backbone of MidasTouch is a Monte-Carlo particle filter, with a measurement model based on a tactile code networ…
University of Liverpool（利物浦大学）；smARTLab（smARTLab实验室）
Tactile sensing is important for robots to perceive the world as it captures the texture and hardness of the object in contact and is robust to illumination and colour variances. However, due to the limited sensing area and the resistance of the fixed surface, current tactile sensors have to tap the tactile sensor on target object many times when assessing a large surface, i.e., pressing, lifting up and shifting to another region. This process is ineffective and time consuming. It is also undesirable to drag such sensors as this often damages the sensitive membrane of the sensor or the object.…
Bristol Robotics Laboratory（布里斯托机器人实验室）；University of Bristol（布里斯托大学）
Our human sense of touch enables us to manipulate our surroundings; therefore, complex robotic manipulation will require artificial tactile sensing. Typically tactile sensor arrays are used in robotics, implying that a straightforward way of interpreting multidimensional data is required. In this paper we present a simple visualisation approach based on applying principal component analysis (PCA) to systematically collected sets of tactile data. We apply the visualisation approach to 4 different types of tactile sensor, encompassing fingertips and vibrissal arrays. The results show that PCA ca…
TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
Shanghai Jiao Tong University（上海交通大学）；Xense Robotics（Xense机器人公司）；Sun Yat-sen University（中山大学）；Oxford（牛津大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；UCSD（加州大学圣地亚哥分校）
Tactile-WAM: Touch-Aware World Action Model with Tactile Asymmetric Attention
Ant Group（蚂蚁集团）；Zhejiang University（浙江大学）；The University of Hong Kong（香港大学）；Shenzhen Loop Area Institute（深圳河套学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
University of Leeds（利兹大学）；Istituto Italiano di Tecnologia（意大利理工学院）；University of Glasgow（格拉斯哥大学）
Touch sensing can help robots understand their sur- rounding environment, and in particular the objects they interact with. To this end, roboticists have, in the last few decades, developed several tactile sensing solutions, extensively reported in the literature. Research into interpreting the conveyed tactile information has also started to attract increasing attention in recent years. However, a comprehensive study on this topic is yet to be reported. In an effort to collect and summarize the major scientific achievements in the area, this survey extensively reviews current trends in robot …
Model-Based Capacitive Touch Sensing in Soft Robotics: Achieving Robust Tactile Interactions for Artistic Applications
Departamento de Mecánica de la Universidad de Chile（智利大学机械系）；Soft Robotics Lab (SRL) at ETH Zurich（苏黎世联邦理工学院软机器人实验室）；University of Applied Sciences Karlsruhe（卡尔斯鲁厄应用技术大学）；Proxception GmbH（Proxception有限公司）
This work investigates uncertainty-aware deep learning (DL) in tactile robotics based on a general framework introduced recently for robot vision. For a test scenario, we consider optical tactile sensing in combination with DL to estimate the edge pose as a feedback signal to servo around various 2D test objects. We demonstrate that uncertainty-aware DL can improve the pose estimation over deterministic DL methods. The system estimates the uncertainty associated with each prediction, which is used along with temporal coherency to improve the predictions via a Kalman filter, and hence improve t…
Improbable AI Lab（Improbable人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）
We study the problem of object retrieval in scenarios where visual sensing is absent, object shapes are unknown beforehand and objects can move freely, like grabbing objects out of a drawer. Successful solutions require localizing free objects, identifying specific object instances, and then grasping the identified objects, only using touch feedback. Unlike vision, where cameras can observe the entire scene, touch sensors are local and only observe parts of the scene that are in contact with the manipulator. Moreover, information gathering via touch sensors necessitates applying forces on the …
For robot touch to converge with the human sense of touch, artificial transduction should involve biologically-plausible population codes analogous to those of natural afferents. Using a biomimetic tactile sensor with 3d-printed skin based on the dermal-epidermal boundary, we propose two novel feature sets to mimic slowly-adapting and rapidly-adapting type-I tactile mechanoreceptor function. Their plausibility is tested with three classic experiments from the study of natural touch: impingement on a flat plate to probe adaptation and spatial modulation; stimulation by spatially-complex ridged …
TouchDrive: Electronics-Free Tactile Sensing Interface for Assistive Grasping
Division of Microsystems Technology, Department of Electrical Engineering（微系统技术部门）；Cyber-Physical Systems Lab, Department of Information Technology（网络物理系统实验室）
TacGen: Touch Is a Necessary Dimension of Physical-World Representation -- Addressing Tactile Data Scarcity with Scalable Vision-to-Touch Alignment and Generation
University of Maryland, College Park（马里兰大学帕克分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Alabama（阿拉巴马大学）；University of Washington（华盛顿大学）；Stanford University（斯坦福大学）
ViTacGen: Robotic Pushing with Vision-to-Touch Generation
Department of Engineering, King’s College London（工程系，伦敦国王学院）；Department of Engineering Mathematics and Bristol, Robotics Laboratory, University of Bristol（工程数学系及布里斯托尔机器人实验室，布里斯托尔大学）
