We propose a method that simultaneously estimates and controls extrinsic contact with tactile feedback. The method enables challenging manipulation tasks that require controlling light forces and accurate motions in contact, such as balancing an unknown object on a thin rod standing upright. A factor graph-based framework fuses a sequence of tactile and kinematic measurements to estimate and control the interaction between gripper-object-environment, including the location and wrench at the extrinsic contact between the grasped object and the environment and the grasp wrench transferred from the gripper to the object. The same framework simultaneously plans the gripper motions that make it possible to estimate the state while satisfying regularizing control objectives to prevent slip, such as minimizing the grasp wrench and minimizing frictional force at the extrinsic contact. We show results with sub-millimeter contact localization error and good slip prevention even on slippery environments, for multiple contact formations (point, line, patch contact) and transitions between them. See supplementary video and results at https://sites.google.com/view/sim-tact.
This paper addresses the localization of contacts of an unknown grasped rigid object with its environment, i.e., extrinsic to the robot. We explore the key role that distributed tactile sensing plays in localizing contacts external to the robot, in contrast to the role that aggregated force/torque measurements play in localizing contacts on the robot. When in contact with the environment, an object will move in accordance with the kinematic and possibly frictional constraints imposed by that contact. Small motions of the object, which are observable with tactile sensors, indirectly encode thos…
We present Neural Contact Fields, a method that brings together neural fields and tactile sensing to address the problem of tracking extrinsic contact between object and environment. Knowing where the external contact occurs is a first step towards methods that can actively control it in facilitating downstream manipulation tasks. Prior work for localizing environmental contacts typically assume a contact type (e.g. point or line), does not capture contact/no-contact transitions, and only works with basic geometric-shaped objects. Neural Contact Fields are the first method that can track arbit…
We propose a method that actively estimates contact location between a grasped rigid object and its environment and uses this as input to a peg-in-hole insertion policy. An estimation model and an active tactile feedback controller work collaboratively to estimate the external contacts accurately. The controller helps the estimation model get a better estimate by regulating a consistent contact mode. The better estimation makes it easier for the controller to regulate the contact. We then train an object-agnostic insertion policy that learns to use the series of contact estimates to guide the …
Tactile Estimation of Extrinsic Contact Patch for Stable Placement
Mitsubishi Electric Corporation（三菱电机株式会社）；Mitsubishi Electric Research Labs（三菱电机研究所）；Tokyo Institute of Technology（东京工业大学）；Massachusetts Institute of Technology（麻省理工学院）
National University of Singapore（新加坡国立大学）；Guilin University of Electronic Technology（桂林电子科技大学）
The tactile sensing capabilities of human hands are essential in performing daily activities. Simultaneously perceiving normal and shear forces via the mechanoreceptors integrated into the hands enables humans to achieve daily tasks like grasping delicate objects. In this paper, we design and fabricate a novel biomimetic tactile sensor with skin-like heterogeneity that perceives normal and shear contact forces simultaneously. It mimics the multilayers of mechanoreceptors by combining an extrinsic layer (piezoresistive sensors) and an intrinsic layer (a Hall sensor) so that it can perform estim…
Joint estimation of grasped object pose and extrinsic contacts is central to robust and dexterous manipulation. In this paper, we propose a novel state-estimation algorithm that jointly estimates contact location and object pose in 3D using exclusively proprioception and tactile feedback. Our approach leverages two complementary particle filters: one to estimate contact location (CPFGrasp) and another to estimate object poses (SCOPE). We implement and evaluate our approach on real-world single-arm and dual-arm robotic systems. We demonstrate that by bringing two objects into contact, the robot…
University of Birmingham（伯明翰大学）；Chalmers University of Technology（查尔姆斯理工大学）；University College London（伦敦大学学院）
This paper addresses the problem of simultaneously exploring an unknown object to model its shape, using tactile sensors on robotic fingers, while also improving finger placement to optimise grasp stability. In many situations, a robot will have only a partial camera view of the near side of an observed object, for which the far side remains occluded. We show how an initial grasp attempt, based on an initial guess of the overall object shape, yields tactile glances of the far side of the object which enable the shape estimate and consequently the successive grasps to be improved. We propose a …
Placing by Touching: An empirical study on the importance of tactile sensing for precise object placing
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息技术研究所）；Bielefeld University（比勒费尔德大学）；PAL Robotics（PAL机器人公司）；Technical University Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science（认知科学中心）
This work details the design of a novel two finger robot gripper with multiple Gelsight based optical-tactile sensors covering the inner surface of the hand. The multiple Gelsight sensors can gather the surface topology of the object from multiple views simultaneously as well as can track the shear and tensile stress. In addition, other sensing modalities enable the hand to gather the thermal, acoustic and vibration information from the object being grasped. The force controlled gripper is fully actuated so that it can be used for various grasp configurations and can also be used for in-hand m…
University of Bristol（布里斯托大学）；University of Pisa（比萨大学）；Istituto Italiano di Tecnologia（意大利技术研究院）
This paper presents a control scheme for force sensitive, gentle grasping with a Pisa/IIT anthropomorphic SoftHand equipped with a miniaturised version of the TacTip optical tactile sensor on all five fingertips. The tactile sensors provide high-resolution information about a grasp and how the fingers interact with held objects. We first describe a series of hardware developments for performing asynchronous sensor data acquisition and processing, resulting in a fast control loop sufficient for real-time grasp control. We then develop a novel grasp controller that uses tactile feedback from all…
Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Beijing Key Lab of Behavior and Mental Health, Peking University（北京大学行为与心理健康北京市重点实验室）；Beijing Institute for General Artificial Intelligence（北京一般人工智能研究院）；State Key Lab for General Artificial Intelligence（一般人工智能国家重点实验室）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（具身智能实验室，北京大学武汉人工智能研究院）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）
TacFinRay: Soft Tactile Fin-Ray Finger with Indirect Tactile Sensing for Robust Grasping
Graduate School of Data Science at Kyungpook National University（Kyungpook National University 数据科学研究生院）；University of Bristol（布里斯托大学）；Bristol Robotics Laboratory（布里斯托机器人实验室）
RoTipBot: Robotic Handling of Thin and Flexible Objects using Rotatable Tactile Sensors
School of Aerospace Engineering, Beijing Institute of Technology（北京理工大学航天工程学院）；Department of Engineering, King’s College London（伦敦国王学院工程系）；Department of Data Science and AI, Monash University（墨尔本大学数据科学与人工智能系）
Aalto University（阿尔托大学）；University of Bristol（布里斯托大学）；University of the West of England（西英格兰大学）
RatSLAM is a rat hippocampus-inspired visual Simultaneous Localization and Mapping (SLAM) framework capable of generating semi-metric topological representations of indoor and outdoor environments. Whisker-RatSLAM is a 6D extension of the RatSLAM and primarily focuses on object recognition by generating point clouds of objects based on whisking information. This paper introduces a novel extension to both former works that is referred to as ViTa-SLAM that harnesses both vision and tactile information for performing SLAM. This not only allows the robot to perform natural interaction with the env…
DTactive: A Vision-Based Tactile Sensor with Active Surface
Shanghai Qi Zhi Institute（上海期智研究院）；Huazhong University of Science and Technology（华中科技大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）
MagicGel: A Novel Visual-Based Tactile Sensor Design with MagneticGel
School of Mechanical Engineering, Anhui University of Technology（安徽工业大学机械工程学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）；College of Quality and Technical Supervision, Hebei University（河北大学质量技术监督学院）；School of Engineering and Technology, China University of Geosciences (Beijing)（中国地质大学（北京）工程技术学院）
Tactile sensing is an essential capability for robots that carry out dexterous manipulation tasks. While cameras, Lidars and other remote sensors can assess a scene globally and instantly, tactile sensors can reduce their measurement uncertainties and gain information about the local physical interactions between the in-contact objects and the robot, that are often not accessible via remote sensing. Tactile sensors can be grouped into two main categories: electronic tactile skins and camera based optical tactile sensors. The former are slim and can be fitted to different body parts, whereas th…
CompdVision: Combining Near-Field 3D Visual and Tactile Sensing Using a Compact Compound-Eye Imaging System
The Hong Kong University of Science and Technology（香港科技大学）；Great Bay University（大湾区大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳香港协同创新研究院）
TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot
Southern University of Science and Technology（南方科技大学）；University of Hong Kong（香港大学）；Tencent Robotics X（腾讯 Robotics X）；University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）
FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation
Shanghai Innovation Institute（上海创新研究院）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；Shanghai University（上海大学）
University of California, Berkeley（加利福尼亚大学伯克利分校）；Harbin Institute of Technology（哈尔滨工业大学）；Research Institute of Intelligent Control and Systems, Harbin Institute of Technology（哈尔滨工业大学智能控制与系统研究所）；Department of Mechanical Engineering, University of California, Berkeley（加利福尼亚大学伯克利分校机械工程系）
As more robots are implemented for contact-rich tasks, tactile sensors are in increasing demand. For many circumstances, the contact is required to be compliant, and soft sensors are in need. This paper introduces a novelly designed soft sensor that can simultaneously estimate the contact force and contact location. Inspired by humans' skin, which contains multi-layers of receptors, the designed tactile sensor has a dual-layer structure. The first layer is made of a conductive fabric that is responsible for sensing the contact force. The second layer is composed of four small conductive rubber…
The University of Hong Kong（香港大学）；CNRS-University of Montpellier（法国国家科学研究中心-蒙彼利埃大学）；Centre for Transformative Garment Production（转化制衣生产中心）
Human skin can accurately sense the self-decoupled normal and shear forces when in contact with objects of different sizes. Although there exist many soft and conformable tactile sensors on robotic applications able to decouple the normal force and shear forces, the impact of the size of object in contact on the force calibration model has been commonly ignored. Here, using the principle that contact force can be derived from the light power loss in the soft optical fiber core, we present a soft tactile sensor that decouples normal and shear forces and calibrates the measurement results based …
TacMan-Turbo: Proactive Tactile Control for Robust and Efficient Articulated Object Manipulation
Peking University（北京大学）；State Key Lab of General AI at Peking University（北京大学通用人工智能国家重点实验室）；PKU-BingJi Joint Laboratory for Artificial Intelligence（北京大学-百度智云人工智能联合实验室）；Wuhan Major Scientific and Technological Special Program（武汉重大科技专项）；Hubei Embodied Intelligence Foundation Model Research and Development Program（湖北省具身智能基础模型研发计划）；National Comprehensive Experimental Base for Governance of Intelligent Society（智能社会治理国家综合实验基地）；LeapZenith AI Research（LeapZenith人工智能研究）；Tsinghua University（清华大学）；Embodied Intelligence Lab（具身智能实验室）；PKU-Wuhan Institute for Artificial Intelligence（北京大学-武汉人工智能研究院）；University of Chinese Academy of Sciences（中国科学院大学）
MagCilia: A Compact Magnetociliary Tactile Sensor with 3D Force Sensing for Robotic Contact Perception and Grasping Feedback
National University of Singapore（新加坡国立大学）；University of California, Berkeley（加州大学伯克利分校）；City University of Hong Kong（香港城市大学）；City University of Hong Kong Shenzhen Research Institute（香港城市大学深圳研究院）
FasTac: A Curved Multispectral Vision-Based Tactile Sensor for High-Speed High-Precision 3D Shape and Force Perception
State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology（华中科技大学机械科学与工程学院智能制造装备与技术国家重点实验室）；National Innovation Institute of Digital Design and Manufacturing（国家数字化设计与制造创新中心）
Industrial robots are increasingly used in various applications where the robot accuracy becomes very important, hence calibrations of the robot's kinematic parameters and the measurement system's extrinsic parameters are required. However, the existing calibration approaches are either too cumbersome or require another expensive external measurement system such as laser tracker or measurement spinarm. In this paper, we propose SCALAR, a calibration method to simultaneously improve the kinematic parameters of a 6-DoF robot and the extrinsic parameters of a 2D Laser Range Finder (LRF) which is …
