In robots, nonprehensile manipulation operations such as pushing are a useful way of moving large, heavy or unwieldy objects, moving multiple objects at once, or reducing uncertainty in the location or pose of objects. In this study, we propose a reactive and adaptive method for robotic pushing that uses rich feedback from a high-resolution optical tactile sensor to control push movements instead of relying on analytical or data-driven models of push interactions. Specifically, we use goal-driven tactile exploration to actively search for stable pushing configurations that cause the object to maintain its pose relative to the pusher while incrementally moving the pusher and object towards the target. We evaluate our method by pushing objects across planar and curved surfaces. For planar surfaces, we show that the method is accurate and robust to variations in initial contact position/angle, object shape and start position; for curved surfaces, the performance is degraded slightly. An immediate consequence of our work is that it shows that explicit models of push interactions might be sufficient but are not necessary for this type of task. It also raises the interesting question of which aspects of the system should be modelled to achieve the best performance and generalization across a wide range of scenarios. Finally, it highlights the importance of testing on non-planar surfaces and in other more complex environments when developing new methods for robotic pushing.
Massachusetts Institute of Technology（麻省理工学院）；Tsinghua University（清华大学）
Soft robots offer significant advantages in adaptability, safety, and dexterity compared to conventional rigid-body robots. However, it is challenging to equip soft robots with accurate proprioception and tactile sensing due to their high flexibility and elasticity. In this work, we describe the development of a vision-based proprioceptive and tactile sensor for soft robots called GelFlex, which is inspired by previous GelSight sensing techniques. More specifically, we develop a novel exoskeleton-covered soft finger with embedded cameras and deep learning methods that enable high-resolution pr…
Xi’an-Jiaotong Liverpool University（西交利物浦大学）；The University of Hong Kong（香港大学）；City University of Hong Kong（香港城市大学）；Southern University of Science and Technology（南方科技大学）
We propose a deep visuo-tactile model for realtime estimation of the liquid inside a deformable container in a proprioceptive way.We fuse two sensory modalities, i.e., the raw visual inputs from the RGB camera and the tactile cues from our specific tactile sensor without any extra sensor calibrations.The robotic system is well controlled and adjusted based on the estimation model in real time. The main contributions and novelties of our work are listed as follows: 1) Explore a proprioceptive way for liquid volume estimation by developing an end-to-end predictive model with multi-modal convolut…
To adapt to constantly changing environments and be safe for human interaction, robots should have compliant and soft characteristics as well as the ability to sense the world around them. Even so, the incorporation of tactile sensing into a soft compliant robot, like the Fin Ray finger, is difficult due to its deformable structure. Not only does the frame need to be modified to allow room for a vision sensor, which enables intricate tactile sensing, the robot must also retain its original mechanically compliant properties. However, adding high-resolution tactile sensors to soft fingers is dif…
Facebook AI Research（Facebook人工智能研究院）；Carnegie Mellon University（卡内基梅隆大学）
We're interested in the problem of estimating object states from touch during manipulation under occlusions. In this work, we address the problem of estimating object poses from touch during planar pushing. Vision-based tactile sensors provide rich, local image measurements at the point of contact. A single such measurement, however, contains limited information and multiple measurements are needed to infer latent object state. We solve this inference problem using a factor graph. In order to incorporate tactile measurements in the graph, we need local observation models that can map high-dime…
Georgia Institute of Technology（佐治亚理工学院）；NVIDIA（英伟达）；University of Utah（犹他大学）；University of Washington（华盛顿大学）
To perform complex tasks, robots must be able to interact with and manipulate their surroundings. One of the key challenges in accomplishing this is robust state estimation during physical interactions, where the state involves not only the robot and the object being manipulated, but also the state of the contact itself. In this work, within the context of planar pushing, we extend previous inference-based approaches to state estimation in several ways. We estimate the robot, object, and the contact state on multiple manipulation platforms configured with a vision-based articulated model track…
NVIDIA（英伟达）；University of Utah（犹他大学）；Georgia Institute of Technology（佐治亚理工学院）；University of Washington（华盛顿大学）
Current methods for estimating force from tactile sensor signals are either inaccurate analytic models or task-specific learned models. In this paper, we explore learning a robust model that maps tactile sensor signals to force. We specifically explore learning a mapping for the SynTouch BioTac sensor via neural networks. We propose a voxelized input feature layer for spatial signals and leverage information about the sensor surface to regularize the loss function. To learn a robust tactile force model that transfers across tasks, we generate ground truth data from three different sources: (1)…
Tactile-Proprioceptive Sensor Fusion for Contact Wrench Estimation in Whole-Body Physical Human-Robot Interaction
Department of Robotics and Mechatronics Engineering, DGIST (Daegu Gyeongbuk Institute of Science and Technology)（机器人与机电工程系，DGIST（大邱庆尚科学技术研究所））；Kinetic Intelligent Machine Lab (KIMLAB), University of Illinois Urbana-Champaign（动能智能机器实验室（KIMLAB），伊利诺伊大学厄巴纳-香槟分校）
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Mechanical Engineering Department, Massachusetts Institute of Technology（麻省理工学院机械工程系）；XYZ Robotics（XYZ机器人公司）；Electrical and Computer Engineering Department, Brigham Young University（杨百翰大学电气与计算机工程系）
Tactile perception is central to robot manipulation in unstructured environments. However, it requires contact, and a mature implementation must infer object models while also accounting for the motion induced by the interaction. In this work, we present a method to estimate both object shape and pose in real-time from a stream of tactile measurements. This is applied towards tactile exploration of an unknown object by planar pushing. We consider this as an online SLAM problem with a nonparametric shape representation. Our formulation of tactile inference alternates between Gaussian process im…
Towards Interpretable Visuo-Tactile Predictive Models for Soft Robot Interactions
The BioRobotics Institute, Sant’Anna School of Advanced Studies（圣安娜高级研究院生物机器人研究所）；Departement of Excellence in Robotics & AI, Sant’Anna School of Advanced Studies（圣安娜高级研究院机器人与人工智能卓越系）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
ViTacGen: Robotic Pushing with Vision-to-Touch Generation
Department of Engineering, King’s College London（工程系，伦敦国王学院）；Department of Engineering Mathematics and Bristol, Robotics Laboratory, University of Bristol（工程数学系及布里斯托尔机器人实验室，布里斯托尔大学）
The recent development of novel aerial vehicles capable of physically interacting with the environment leads to new applications such as contact-based inspection. These tasks require the robotic system to exchange forces with partially-known environments, which may contain uncertainties including unknown spatially-varying friction properties and discontinuous variations of the surface geometry. Finding a control strategy that is robust against these environmental uncertainties remains an open challenge. This paper presents a learning-based adaptive control strategy for aerial sliding tasks. In…
Tencent Robotics X（腾讯机器人X实验室）；Imperial College London（帝国理工学院）
Reinforcement Learning (RL) methods have been widely applied for robotic manipulations via sim-to-real transfer, typically with proprioceptive and visual information. However, the incorporation of tactile sensing into RL for contact-rich tasks lacks investigation. In this paper, we model a tactile sensor in simulation and study the effects of its feedback in RL-based robotic control via a zero-shot sim-to-real approach with domain randomization. We demonstrate that learning and controlling with feedback from tactile sensor arrays at the gripper, both in simulation and reality, can enhance gras…
The University of Melbourne（墨尔本大学）；St. Vincent’s Hospital（圣文森特医院）
Tactile-based blind grasping addresses realistic robotic grasping in which the hand only has access to proprioceptive and tactile sensors. The robotic hand has no prior knowledge of the object/grasp properties, such as object weight, inertia, and shape. There exists no manipulation controller that rigorously guarantees object manipulation in such a setting. Here, a robust control law is proposed for object manipulation in tactile-based blind grasping. The analysis ensures semi-global asymptotic and exponential stability in the presence of model uncertainties and external disturbances that are …
Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation. However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for fine manipulation. Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment. Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action. Grasp…
University of Lincoln（林肯大学）；Politecnico di Milano（米兰理工大学）
This paper introduces a novel approach to address the problem of Physical Robot Interaction (PRI) during robot pushing tasks. The approach uses a data-driven forward model based on tactile predictions to inform the controller about potential future movements of the object being pushed, such as a strawberry stem, using a robot tactile finger. The model is integrated into a Deep Functional Predictive Control (d-FPC) system to control the displacement of the stem on the tactile finger during pushes. Pushing an object with a robot finger along a desired trajectory in 3D is a highly nonlinear and c…
Perceiving the environment through touch is important for robots to reach in cluttered environments, but devising a way to sense without disturbing objects is challenging. This work presents the design and modelling of whisker-inspired sensors that attach to the surface of a robot manipulator to sense its surrounding through light contacts. We obtain a sensor model using a calibration process that applies to straight and curved whiskers. We then propose a sensing algorithm using Bayesian filtering to localize contact points. The algorithm combines the accurate proprioceptive sensing of the rob…
Placing by Touching: An empirical study on the importance of tactile sensing for precise object placing
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息技术研究所）；Bielefeld University（比勒费尔德大学）；PAL Robotics（PAL机器人公司）；Technical University Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science（认知科学中心）
Accurate and robust object state estimation enables successful object manipulation. Visual sensing is widely used to estimate object poses. However, in a cluttered scene or in a tight workspace, the robot's end-effector often occludes the object from the visual sensor. The robot then loses visual feedback and must fall back on open-loop execution. In this paper, we integrate both tactile and visual input using a framework for solving the SLAM problem, incremental smoothing and mapping (iSAM), to provide a fast and flexible solution. Visual sensing provides global pose information but is noisy …
Contact SLAM: An Active Tactile Exploration Policy Based on Physical Reasoning Utilized in Robotic Fine Blind Manipulation Tasks
Research Center for Intelligent Robotics（智能机器人研究中心）；National Key Laboratory of Aerospace Flight Dynamics（航空飞行动力学国家重点实验室）；Shaanxi Province Innovation Team of Intelligent Robotic Technology（陕西省智能机器人技术创新团队）；School of Astronautics, Northwestern Polytechnical University（航空学院，西北工业大学）
Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer \cite{dosovitskiy2021image} to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus at…
King’s College London（伦敦国王学院）；University of Liverpool（利物浦大学）；Queen Mary University of London（伦敦玛丽女王大学）
For humans, both the proprioception and touch sensing are highly utilized when performing haptic perception. However, most approaches in robotics use only either proprioceptive data or touch data in haptic object recognition. In this paper, we present a novel method named Iterative Closest Labeled Point (iCLAP) to link the kinesthetic cues and tactile patterns fundamentally and also introduce its extensions to recognize object shapes. In the training phase, the iCLAP first clusters the features of tactile readings into a codebook and assigns these features with distinct label numbers. A 4D poi…
Distributed Cascade Force Control of Soft-Tactile-Based Multi-robot System for Object Transportation
University of Prince Edward Island（爱德华王子岛大学）；Japan Advanced Institute of Science and Technology（北陆先端科学技术大学院大学）；University of Engineering and Technology, Vietnam（越南工程技术大学）；Japan Science and Technology Agency（日本科学技术振兴机构）
The connection between visual input and tactile sensing is critical for object manipulation tasks such as grasping and pushing. In this work, we introduce the challenging task of estimating a set of tactile physical properties from visual information. We aim to build a model that learns the complex mapping between visual information and tactile physical properties. We construct a first of its kind image-tactile dataset with over 400 multiview image sequences and the corresponding tactile properties. A total of fifteen tactile physical properties across categories including friction, compliance…
Istituto Italiano di Tecnologia（意大利理工学院）；Fakulta Elektrotechnická, ČVUT v Praze（捷克理工大学电气工程学院）
Somatosensory inputs can be grossly divided into tactile (or cutaneous) and proprioceptive -- the former conveying information about skin stimulation, the latter about limb position and movement. The principal proprioceptors are constituted by muscle spindles, which deliver information about muscle length and speed. In primates, this information is relayed to the primary somatosensory cortex and eventually the posterior parietal cortex, where integrated information about body posture (postural schema) is presumably available. However, coming from robotics and seeking a biologically motivated m…
In primate brains, tactile and proprioceptive inputs are relayed to the somatosensory cortex which is known for somatotopic representations, or, "homunculi". Our research centers on understanding the mechanisms of the formation of these and more higher-level body representations (body schema) by using humanoid robots and neural networks to construct models. We specifically focus on how spatial representation of the body may be learned from somatosensory information in self-touch configurations. In this work, we target the representation of proprioceptive inputs, which we take to be joint angle…
Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition
Research Center for Intelligent Robotics, School of Astronautics, Northwestern Polytechnical University（西北工业大学航天学院智能机器人研究中心）；National Key Laboratory of Aerospace Flight Dynamics, Northwestern Polytechnical University（西北工业大学航空航天飞行动力学全国重点实验室）
