Elastic Interaction of Particles for Robotic Tactile Simulation
Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心（BNRist））；State Key Lab on Intelligent Technology and Systems（智能技术与系统国家重点实验室）；Tsinghua University（清华大学）
Tactile sensing plays an important role in robotic perception and manipulation. To overcome the real-world limitations of data collection, simulating tactile response in virtual environment comes as a desire direction of robotic research. Most existing works model the tactile sensor as a rigid multi-body, which is incapable of reflecting the elastic property of the tactile sensor as well as characterizing the fine-grained physical interaction between two objects. In this paper, we propose Elastic Interaction of Particles (EIP), a novel framework for tactile emulation. At its core, EIP models the tactile sensor as a group of coordinated particles, and the elastic theory is applied to regulate the deformation of particles during the contact process. The implementation of EIP is conducted from scratch, without resorting to any existing physics engine. Experiments to verify the effectiveness of our method have been carried out on two applications: robotic perception with tactile data and 3D geometric reconstruction by tactile-visual fusion. It is possible to open up a new vein for robotic tactile simulation, and contribute to various downstream robotic tasks.
Tactile sensing plays an important role in robotic perception and manipulation tasks. To overcome the real-world limitations of data collection, simulating tactile response in a virtual environment comes as a desirable direction of robotic research. In this paper, we propose Elastic Interaction of Particles (EIP) for tactile simulation. Most existing works model the tactile sensor as a rigid multi-body, which is incapable of reflecting the elastic property of the tactile sensor as well as characterizing the fine-grained physical interaction between the two objects. By contrast, EIP models the …
University of Leeds（利兹大学）；Istituto Italiano di Tecnologia（意大利理工学院）；University of Glasgow（格拉斯哥大学）
Touch sensing can help robots understand their sur- rounding environment, and in particular the objects they interact with. To this end, roboticists have, in the last few decades, developed several tactile sensing solutions, extensively reported in the literature. Research into interpreting the conveyed tactile information has also started to attract increasing attention in recent years. However, a comprehensive study on this topic is yet to be reported. In an effort to collect and summarize the major scientific achievements in the area, this survey extensively reviews current trends in robot …
Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京大学行为与心理健康北京市重点实验室）；State Key Lab of General AI, Beijing Institute for General AI（人工智能通用状态关键实验室，北京人工智能研究院）；AIVC Laboratory, University of California, Los Angeles（加州大学洛杉矶分校AIVC实验室）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（北京大学武汉人工智能研究院具身智能实验室）
This paper describes the design of a multi-camera optical tactile sensor that provides information about the contact force distribution applied to its soft surface. This information is contained in the motion of spherical particles spread within the surface, which deforms when subject to force. The small embedded cameras capture images of the different particle patterns that are then mapped to the three-dimensional contact force distribution through a machine learning architecture. The design proposed in this paper exhibits a larger contact surface and a thinner structure than most of the exis…
ETH Zurich（苏黎世联邦理工学院）；Nottingham Trent University（诺丁汉特伦特大学）；Ulster University（阿尔斯特大学）
Robots need to exploit high-quality information on grasped objects to interact with the physical environment. Haptic data can therefore be used for supplementing the visual modality. This paper investigates the use of Convolutional Neural Networks (CNN) and Long-Short Term Memory (LSTM) neural network architectures for object classification on Spatio-temporal tactile grasping data. Furthermore, we compared these methods using data from two different fingertip sensors (namely the BioTac SP and WTS-FT) in the same physical setup, allowing for a realistic comparison across methods and sensors for…
Department of Engineering, King’s College London（伦敦国王学院工程系）；School of Engineering Mathematics and Bristol Robotics Laboratory, University of Bristol（布里斯托大学工程数学学院和布里斯托机器人实验室）；Department of Computer Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校计算机科学系）；School of Engineering and Materials Science, Queen Mary University of London（伦敦Queen Mary大学工程与材料科学学院）；Institute for Cognitive Systems (ICS), Technische Universität München（慕尼黑技术大学认知系统研究所）；Department of Electrical and Computer Engineering, Northeastern University（东北大学电气与计算机工程系）
Technical University of Munich（慕尼黑工业大学）；ABB Corporate Research（ABB集团研究院）；University of Campania Luigi Vanvitelli（坎帕尼亚大学路易吉·万维泰利）；Institute of Robotics and Mechatronics, German Aerospace Center（德国航空航天中心机器人与机电一体化研究所）
In this work, we introduce the problem of cross-modal visuo-tactile object recognition with robotic active exploration. With this term, we mean that the robot observes a set of objects with visual perception and, later on, it is able to recognize such objects only with tactile exploration, without having touched any object before. Using a machine learning terminology, in our application we have a visual training set and a tactile test set, or vice versa. To tackle this problem, we propose an approach constituted by four steps: finding a visuo-tactile common representation, defining a suitable …
In this paper, a novel tactile sensing mechanism for soft robotic fingers is proposed. Inspired by the proprioception mechanism found in mammals, the proposed approach infers tactile information from a strain sensor attached on the finger's tendon. We perform experiments to test the tactile sensing capabilities of the proposed structures, and our results indicate this method is capable of palpating texture and stiffness in both abduction and flexion contact. Under systematic cross validation, the proposed system achieved 100% and 99.7% accuracy in texture and stiffness discrimination respectiv…
Soft optical tactile sensors enable robots to manipulate deformable objects by capturing important features such as high-resolution contact geometry and estimations of object compliance. This work presents a variable stiffness soft tactile end-effector called StRETcH, a Soft to Resistive Elastic Tactile Hand, that is easily manufactured and integrated with a robotic arm. An elastic membrane is suspended between two robotic fingers, and a depth sensor capturing the deformations of the elastic membrane enables sub-millimeter accurate estimates of contact geometries. The parallel-jaw gripper vari…
Teaching dexterity to multi-fingered robots has been a longstanding challenge in robotics. Most prominent work in this area focuses on learning controllers or policies that either operate on visual observations or state estimates derived from vision. However, such methods perform poorly on fine-grained manipulation tasks that require reasoning about contact forces or about objects occluded by the hand itself. In this work, we present T-Dex, a new approach for tactile-based dexterity, that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used to train self…
University of Bristol（布里斯托大学）；Universit di Pisa（比萨大学）；Istituto Italiano di Tecnologia（意大利理工学院）
In this work, we report on the integrated sensorimotor control of the Pisa/IIT SoftHand, an anthropomorphic soft robot hand designed around the principle of adaptive synergies, with the BRL tactile fingertip (TacTip), a soft biomimetic optical tactile sensor based on the human sense of touch. Our focus is how a sense of touch can be used to control an anthropomorphic hand with one degree of actuation, based on an integration that respects the hand's mechanical functionality. We consider: (i) closed-loop tactile control to establish a light contact on an unknown held object, based on the struct…
3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception
State Key Laboratory of Digital Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology（数字制造装备与技术国家重点实验室，机械科学与工程学院，华中科技大学）
Xi’an-Jiaotong Liverpool University（西交利物浦大学）；The University of Hong Kong（香港大学）；City University of Hong Kong（香港城市大学）；Southern University of Science and Technology（南方科技大学）
We propose a deep visuo-tactile model for realtime estimation of the liquid inside a deformable container in a proprioceptive way.We fuse two sensory modalities, i.e., the raw visual inputs from the RGB camera and the tactile cues from our specific tactile sensor without any extra sensor calibrations.The robotic system is well controlled and adjusted based on the estimation model in real time. The main contributions and novelties of our work are listed as follows: 1) Explore a proprioceptive way for liquid volume estimation by developing an end-to-end predictive model with multi-modal convolut…
Robotics Institute（机器人研究所）；The Hong Kong University of Science and Technology（香港科技大学）
In essence, successful grasp boils down to correct responses to multiple contact events between fingertips and objects. In most scenarios, tactile sensing is adequate to distinguish contact events. Due to the nature of high dimensionality of tactile information, classifying spatiotemporal tactile signals using conventional model-based methods is difficult. In this work, we propose to predict and classify tactile signal using deep learning methods, seeking to enhance the adaptability of the robotic grasp system to external event changes that may lead to grasping failure. We develop a deep learn…
This work details the design of a novel two finger robot gripper with multiple Gelsight based optical-tactile sensors covering the inner surface of the hand. The multiple Gelsight sensors can gather the surface topology of the object from multiple views simultaneously as well as can track the shear and tensile stress. In addition, other sensing modalities enable the hand to gather the thermal, acoustic and vibration information from the object being grasped. The force controlled gripper is fully actuated so that it can be used for various grasp configurations and can also be used for in-hand m…
To adapt to constantly changing environments and be safe for human interaction, robots should have compliant and soft characteristics as well as the ability to sense the world around them. Even so, the incorporation of tactile sensing into a soft compliant robot, like the Fin Ray finger, is difficult due to its deformable structure. Not only does the frame need to be modified to allow room for a vision sensor, which enables intricate tactile sensing, the robot must also retain its original mechanically compliant properties. However, adding high-resolution tactile sensors to soft fingers is dif…
Tacchi 2.0: A Low Computational Cost and Comprehensive Dynamic Contact Simulator for Vision-based Tactile Sensors
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；School of Engineering and Technology, China University of Geosciences (Beijing)（中国地质大学（北京）工程技术学院）；School of International, Beijing University of Posts and Telecommunications（北京邮电大学国际学院）；School of Mechanical Engineering, Anhui University of Technology（安徽工业大学机械工程学院）；Zhili College, Tsinghua University（清华大学志道书院）
Representing Data in Robotic Tactile Perception -- A Review
Oxford Robotics Institute (ORI), University of Oxford, United Kingdom（牛津大学机器人研究所（ORI））；Eindhoven University of Technology (TU/e), Netherlands（埃因霍温理工大学（TU/e））；AI, Robotics, & Cognitive Vehicel Lab, BMW Group, Munich, Germany（人工智能、机器人与认知车辆实验室，宝马集团）；Department of Informatics, Bioengineering, Robotics and Systems Engineering (DIBRIS), University of Genoa, Genoa, Italy（信息学、生物工程、机器人与系统工程系（DIBRIS），热那亚大学）
Massachusetts Institute of Technology（麻省理工学院）；Lightspeed & Quantum Studios, Tencent America（腾讯美国光速量子工作室）
Traditional robotic manipulator design methods require extensive, time-consuming, and manual trial and error to produce a viable design. During this process, engineers often spend their time redesigning or reshaping components as they discover better topologies for the robotic manipulator. Tactile sensors, while useful, often complicate the design due to their bulky form factor. We propose an integrated design pipeline to streamline the design and manufacturing of robotic manipulators with knitted, glove-like tactile sensors. The proposed pipeline allows a designer to assemble a collection of …
Embedding high-resolution touch across robotic hands enables adaptive human-like grasping
Institute for Artificial Intelligence（人工智能研究所）；Peking University（北京大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；PKU-Wuhan Institute for Artificial Intelligence（北大武汉人工智能研究所）；College of Engineering（工程学院）；School of Engineering and Materials Science（工程与材料科学学院）
A Biomimetic Fingerprint for Robotic Tactile Sensing
Faculty of Mechatronic \& Micro-Mechatronic Systems Hochschule Karlsruhe (HKA) Karlsruhe, Germany；Deutsches Forschungszentrum für Künstliche Intelligenz GmbH；L3S Research Center, Leibniz Universität Hannover Hannover, Germany
Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation
Queen Mary University of London（伦敦玛丽女王大学）；School of Engineering and Materials Science（工程与材料科学学院）；Humanoid AI（人形人工智能公司）；King’s College London（伦敦国王学院）；School of Biomedical Engineering and Imaging Sciences（生物医学工程与影像科学学院）
Japan Advanced Institute of Science and Technology (JAIST)（日本北陆先端科学技术大学院大学（JAIST））；Hanoi University of Industry（河内工业大学）；Japan Science and Technology Agency（日本科学技术振兴机构）
Robots have been brought to work close to humans in many scenarios. For coexistence and collaboration, robots should be safe and pleasant for humans to interact with. To this end, the robots could be both physically soft with multimodal sensing/perception, so that the robots could have better awareness of the surrounding environment, as well as to respond properly to humans' action/intention. This paper introduces a novel soft robotic link, named ProTac, that possesses multiple sensing modes: tactile and proximity sensing, based on computer vision and a functional material. These modalities co…
In robots, nonprehensile manipulation operations such as pushing are a useful way of moving large, heavy or unwieldy objects, moving multiple objects at once, or reducing uncertainty in the location or pose of objects. In this study, we propose a reactive and adaptive method for robotic pushing that uses rich feedback from a high-resolution optical tactile sensor to control push movements instead of relying on analytical or data-driven models of push interactions. Specifically, we use goal-driven tactile exploration to actively search for stable pushing configurations that cause the object to …
Literary Emotions in Motion: A Soft Robotics Installation for Tactile Storytelling
Department of Mechanical Engineering, University of Chile（智利大学机械工程系）；Independent Researcher（独立研究员）；Bolivian Catholic University（玻利维亚天主大学）；Institute of Engineering Sciences, University of O’Higgins（奥希金斯大学工程科学研究所）
MagCilia: A Compact Magnetociliary Tactile Sensor with 3D Force Sensing for Robotic Contact Perception and Grasping Feedback
National University of Singapore（新加坡国立大学）；University of California, Berkeley（加州大学伯克利分校）；City University of Hong Kong（香港城市大学）；City University of Hong Kong Shenzhen Research Institute（香港城市大学深圳研究院）
Model-Based Capacitive Touch Sensing in Soft Robotics: Achieving Robust Tactile Interactions for Artistic Applications
Departamento de Mecánica de la Universidad de Chile（智利大学机械系）；Soft Robotics Lab (SRL) at ETH Zurich（苏黎世联邦理工学院软机器人实验室）；University of Applied Sciences Karlsruhe（卡尔斯鲁厄应用技术大学）；Proxception GmbH（Proxception有限公司）
M3D-skin: Multi-material 3D-printed Tactile Sensor with Hierarchical Infill Structures for Pressure Sensing
Department of Mechano-Informatics, Graduate School of Information Science and Technology, The University of Tokyo（机械信息学系，信息科学和技术研究生院，东京大学）；AI Center, Graduate School of Information Science and Technology, The University of Tokyo（人工智能中心，信息科学和技术研究生院，东京大学）
University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
Reproducing the capabilities of the human sense of touch in machines is an important step in enabling robot manipulation to have the ease of human dexterity. A combination of robotic technologies will be needed, including soft robotics, biomimetics and the high-resolution sensing offered by optical tactile sensors. This combination is considered here as a SoftBOT (Soft Biomimetic Optical Tactile) sensor. This article reviews the BRL TacTip as a prototypical example of such a sensor. Topics include the relation between artificial skin morphology and the transduction principles of human touch, t…
RoTipBot: Robotic Handling of Thin and Flexible Objects using Rotatable Tactile Sensors
School of Aerospace Engineering, Beijing Institute of Technology（北京理工大学航天工程学院）；Department of Engineering, King’s College London（伦敦国王学院工程系）；Department of Data Science and AI, Monash University（墨尔本大学数据科学与人工智能系）
Tactile sensing is a crucial perception mode for robots and human amputees in need of controlling a prosthetic device. Today robotic and prosthetic systems are still missing the important feature of accurate tactile sensing. This lack is mainly due to the fact that the existing tactile technologies have limited spatial and temporal resolution and are either expensive or not scalable. In this paper, we present the design and the implementation of a hardware-software embedded system called SmartHand. It is specifically designed to enable the acquisition and the real-time processing of high-resol…
In-Hand Object Pose Estimation via Visual-Tactile Fusion
Department of Computer Science, Goethe Universität Frankfurt, Germany（弗赖堡大学计算机科学系）；Intelligent Autonomous Systems Lab, Department of Computer Science, TU Darmstadt, Germany（图恩-达姆斯塔德大学计算机科学系智能自主系统实验室）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, TU Darmstadt（图恩-达姆斯塔德大学认知科学中心）；Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt（黑森人工智能中心（Hessian.AI），达姆斯塔特）
SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects (Early Version)
Tuojing Intelligence（拓景智能）；Tsinghua University（清华大学）；King’s College London（伦敦国王学院）；Southeast University（东南大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Manchester（曼彻斯特大学）；Simple AI（简单人工智能公司）；Imperial College London（伦敦帝国学院）；Carnegie Mellon University（卡内基梅隆大学）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）；The University of Hong Kong（香港大学）
NVIDIA Corporation（英伟达公司）；University of Washington（华盛顿大学）
High-density afferents in the human hand have long been regarded as essential for human grasping and manipulation abilities. In contrast, robotic tactile sensors are typically used to provide low-density contact data, such as center-of-pressure and resultant force. Although useful, this data does not exploit the rich information content that some tactile sensors (e.g., the SynTouch BioTac) naturally provide. This research extends robotic tactile sensing beyond reduced-order models through 1) the automated creation of a precise tactile dataset for the BioTac over diverse physical interactions, …
National University of Singapore（新加坡国立大学）；Guilin University of Electronic Technology（桂林电子科技大学）
The tactile sensing capabilities of human hands are essential in performing daily activities. Simultaneously perceiving normal and shear forces via the mechanoreceptors integrated into the hands enables humans to achieve daily tasks like grasping delicate objects. In this paper, we design and fabricate a novel biomimetic tactile sensor with skin-like heterogeneity that perceives normal and shear contact forces simultaneously. It mimics the multilayers of mechanoreceptors by combining an extrinsic layer (piezoresistive sensors) and an intrinsic layer (a Hall sensor) so that it can perform estim…
University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
Developing artificial tactile sensing capabilities that rival human touch is a long-term goal in robotics and prosthetics. Gradually more elaborate biomimetic tactile sensors are being developed and applied to grasping and manipulation tasks to help achieve this goal. Here we present the neuroTac, a novel neuromorphic optical tactile sensor. The neuroTac combines the biomimetic hardware design from the TacTip sensor which mimicks the layered papillae structure of human glabrous skin, with an event-based camera (DAVIS240, iniVation) and algorithms which transduce contact information in the form…
