Simulators perform an important role in prototyping, debugging, and benchmarking new advances in robotics and learning for control. Although many physics engines exist, some aspects of the real world are harder than others to simulate. One of the aspects that have so far eluded accurate simulation is touch sensing. To address this gap, we present TACTO - a fast, flexible, and open-source simulator for vision-based tactile sensors. This simulator allows to render realistic high-resolution touch readings at hundreds of frames per second, and can be easily configured to simulate different vision-based tactile sensors, including DIGIT and OmniTact. In this paper, we detail the principles that drove the implementation of TACTO and how they are reflected in its architecture. We demonstrate TACTO on a perceptual task, by learning to predict grasp stability using touch from 1 million grasps, and on a marble manipulation control task. Moreover, we provide a proof-of-concept that TACTO can be successfully used for Sim2Real applications. We believe that TACTO is a step towards the widespread adoption of touch sensing in robotic applications, and to enable machine learning practitioners interested in multi-modal learning and control. TACTO is open-source at https://github.com/facebookresearch/tacto.
Improbable AI Lab（Improbable人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）
We study the problem of object retrieval in scenarios where visual sensing is absent, object shapes are unknown beforehand and objects can move freely, like grabbing objects out of a drawer. Successful solutions require localizing free objects, identifying specific object instances, and then grasping the identified objects, only using touch feedback. Unlike vision, where cameras can observe the entire scene, touch sensors are local and only observe parts of the scene that are in contact with the manipulator. Moreover, information gathering via touch sensors necessitates applying forces on the …
Tacchi 2.0: A Low Computational Cost and Comprehensive Dynamic Contact Simulator for Vision-based Tactile Sensors
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；School of Engineering and Technology, China University of Geosciences (Beijing)（中国地质大学（北京）工程技术学院）；School of International, Beijing University of Posts and Telecommunications（北京邮电大学国际学院）；School of Mechanical Engineering, Anhui University of Technology（安徽工业大学机械工程学院）；Zhili College, Tsinghua University（清华大学志道书院）
SimTac: A Physics-Based Simulator for Vision-Based Tactile Sensing with Biomorphic Structures
Department of Engineering, King’s College London（工程系，伦敦国王学院）；School of Aerospace Engineering, Beijing Institute of Technology（航空工程学院，北京理工大学）；Department of Computer Science, University College London（计算机科学系，伦敦大学学院）
We investigate how high-resolution tactile sensors can be utilized in combination with vision and depth sensing, to improve grasp stability prediction. Recent advances in simulating high-resolution tactile sensing, in particular the TACTO simulator, enabled us to evaluate how neural networks can be trained with a combination of sensing modalities. With the large amounts of data needed to train large neural networks, robotic simulators provide a fast way to automate the data collection process. We expand on the existing work through an ablation study and an increased set of objects taken from t…
Facebook AI Research（Facebook人工智能研究院）；McGill University（麦吉尔大学）；University of California, Berkeley（加州大学伯克利分校）
Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. However, in 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings, leaving the active exploration of the shape largely unexplored. Inactive touch sensing for 3D reconstruction, the goal is to actively select the tactile readings that maximize the improvement in shape reconstruction accuracy. However, the development of deep learning-based active touch models is largely limit…
We present BubbleTouch, an open source quasi-static simulator for robotic tactile skins. BubbleTouch can be used to simulate contact with a robot's tactile skin patches as it interacts with humans and objects. The simulator creates detailed traces of contact forces that can be used in experiments in tactile contact activities. We summarize the design of BubbleTouch and highlight our recent work that uses BubbleTouch for experiments with tactile object recognition.
The majority of artificial intelligence research, as it relates from which to biological senses has been focused on vision. The recent explosion of machine learning and in particular, dee p learning, can be partially attributed to the release of high quality data sets for algorithm s from which to model the world on. Thus, most of these datasets are comprised of images. We believe that focusing on sensorimotor systems and tactile feedback will create algorithms that better mimic human intelligence. Here we present SenseNet: a collection of tactile simulators and a large scale dataset of 3D obj…
FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills
School of Automation, Southeast University（东南大学自动化学院）；Key Laboratory of Measurement and Control of CSE, Ministry of Education（东南大学复杂工程系统测量与控制教育部重点实验室）；Department of Engineering, King’s College London（伦敦国王学院工程系）
Robot simulation has been an essential tool for data-driven manipulation tasks. However, most existing simulation frameworks lack either efficient and accurate models of physical interactions with tactile sensors or realistic tactile simulation. This makes the sim-to-real transfer for tactile-based manipulation tasks still challenging. In this work, we integrate simulation of robot dynamics and vision-based tactile sensors by modeling the physics of contact. This contact model uses simulated contact forces at the robot's end-effector to inform the generation of realistic tactile outputs. To el…
TacGen: Touch Is a Necessary Dimension of Physical-World Representation -- Addressing Tactile Data Scarcity with Scalable Vision-to-Touch Alignment and Generation
University of Maryland, College Park（马里兰大学帕克分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Alabama（阿拉巴马大学）；University of Washington（华盛顿大学）；Stanford University（斯坦福大学）
The potential of large tactile arrays to improve robot perception for safe operation in human-dominated environments and of high-resolution tactile arrays to enable human-level dexterous manipulation is well accepted. However, the increase in the number of tactile sensing elements introduces challenges including wiring complexity, power consumption, and data processing. To help address these challenges, we previously developed a tactile sensing technique based compressed sensing that reduces hardware complexity and data transmission, while allowing accurate reconstruction of the full-resolutio…
NVIDIA Corporation（英伟达公司）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
Tactile sensing is critical for robotic grasping and manipulation of objects under visual occlusion. However, in contrast to simulations of robot arms and cameras, current simulations of tactile sensors have limited accuracy, speed, and utility. In this work, we develop an efficient 3D finite element method (FEM) model of the SynTouch BioTac sensor using an open-access, GPU-based robotics simulator. Our simulations closely reproduce results from an experimentally-validated model in an industry-standard, CPU-based simulator, but at 75x the speed. We then learn latent representations for simulat…
Most current works in Sim2Real learning for robotic manipulation tasks leverage camera vision that may be significantly occluded by robot hands during the manipulation. Tactile sensing offers complementary information to vision and can compensate for the information loss caused by the occlusion. However, the use of tactile sensing is restricted in the Sim2Real research due to no simulated tactile sensors being available. To mitigate the gap, we introduce a novel approach for simulating a GelSight tactile sensor in the commonly used Gazebo simulator. Similar to the real GelSight sensor, the sim…
University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
High-resolution optical tactile sensors are increasingly used in robotic learning environments due to their ability to capture large amounts of data directly relating to agent-environment interaction. However, there is a high barrier of entry to research in this area due to the high cost of tactile robot platforms, specialised simulation software, and sim-to-real methods that lack generality across different sensors. In this letter we extend the Tactile Gym simulator to include three new optical tactile sensors (TacTip, DIGIT and DigiTac) of the two most popular types, Gelsight-style (image-sh…
School of Engineering Mathematics and Technology, University of Bristol（工程数学与技术学院，布里斯托尔大学）；Bristol Robotics Laboratory, University of Bristol（布里斯托尔机器人实验室，布里斯托尔大学）
This paper aims to show that robots equipped with a vision-based tactile sensor can perform dynamic manipulation tasks without prior knowledge of all the physical attributes of the objects to be manipulated. For this purpose, a robotic system is presented that is able to swing up poles of different masses, radii and lengths, to an angle of 180 degrees, while relying solely on the feedback provided by the tactile sensor. This is achieved by developing a novel simulator that accurately models the interaction of a pole with the soft sensor. A feedback policy that is conditioned on a sensory obser…
Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京大学行为与心理健康北京市重点实验室）；State Key Lab of General AI, Beijing Institute for General AI（人工智能通用状态关键实验室，北京人工智能研究院）；AIVC Laboratory, University of California, Los Angeles（加州大学洛杉矶分校AIVC实验室）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（北京大学武汉人工智能研究院具身智能实验室）
DOT-Sim: Differentiable Optical Tactile Simulation with Precise Real-to-Sim Physical Calibration
Stanford University（斯坦福大学）；University of Cambridge（剑桥大学）；Toyota Research Institute University 2.0 Program（丰田研究院大学2.0计划）；ARL grant W911NF-21-2-0104（ARL授予的W911NF-21-2-0104项目）；Vannevar Bush Faculty Fellowship（范内var布什教职 fellowship）；Flexiv corporation（Flexiv公司）；Shanghai Jiao Tong University（上海交通大学）
