We address the problem of tracking 3D object poses from touch during in-hand manipulations. Specifically, we look at tracking small objects using vision-based tactile sensors that provide high-dimensional tactile image measurements at the point of contact. While prior work has relied on a-priori information about the object being localized, we remove this requirement. Our key insight is that an object is composed of several local surface patches, each informative enough to achieve reliable object tracking. Moreover, we can recover the geometry of this local patch online by extracting local surface normal information embedded in each tactile image. We propose a novel two-stage approach. First, we learn a mapping from tactile images to surface normals using an image translation network. Second, we use these surface normals within a factor graph to both reconstruct a local patch map and use it to infer 3D object poses. We demonstrate reliable object tracking for over $100$ contact sequences across unique shapes with four objects in simulation and two objects in the real-world. Supplementary video: https://youtu.be/FHks--haOGY
TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);OpenDriveLab at The University of Hong Kong(香港大学OpenDriveLab);Shanghai Jiao Tong University(上海交通大学);East China University of Science and Technology(华东理工大学)
TacSE3: Equivariant SE(3) Motion Estimation from Low-Texture Visuotactile Images for In-Gripper Tracking and Compensation
Hong Kong Center for Construction Robotics, The Hong Kong University of Science and Technology, Hong Kong, China(香港建设机器人中心,香港科技大学,中国香港);The Hong Kong University of Science and Technology (Guangzhou), China(香港科技大学(广州),中国);School of Engineering, Great Bay University, Dongguan, China(工程学院,大湾大学,中国东莞)
Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation
Department of Computer Science, Technical University of Darmstadt(德累斯顿技术大学计算机科学系);German Research Center for AI (DFKI)(德国人工智能研究中心);Centre for Cognitive Science, Technical University of Darmstadt(德累斯顿技术大学认知科学中心);Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt(黑森人工智能中心(Hessian.AI));Robotics Institute Germany (RIG)(德国机器人研究所)
Look-to-Touch: A Vision-Enhanced Proximity and Tactile Sensor for Distance and Geometry Perception in Robotic Manipulation
State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University(上海交通大学机械系统与振动国家重点实验室);Robotics Institute, School of Mechanical Engineering, Shanghai Jiao Tong University(上海交通大学机械工程学院机器人研究所)
Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);Beijing Key Lab of Behavior and Mental Health, Peking University(北京大学行为与心理健康北京市重点实验室);Beijing Institute for General Artificial Intelligence(北京一般人工智能研究院);State Key Lab for General Artificial Intelligence(一般人工智能国家重点实验室);Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence(具身智能实验室,北京大学武汉人工智能研究院);Department of Computer Science and Technology, University of Cambridge(剑桥大学计算机科学与技术系)
Tactile-based force estimation for interaction control with robot fingers
Institut des sytèmes intelligents et de la robotique, Sorbonne University(索邦大学智能与机器人系统研究所);LIRMM, Montpellier University(蒙彼利埃大学LIRMM实验室);Nantes Université, École Centrale Nantes, CNRS, LS2N(南特大学、南特中央理工学院、法国国家科学研究中心、南特数字科学实验室)
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation
Hessian Centre for Artificial Intelligence(黑森人工智能中心);Technische Universität Dresden(德累斯顿工业大学);The Centre for Tactile Internet with Human-in-the-Loop (CeTI)(以人为中心的触觉互联网中心);Technical University of Darmstadt(达姆施塔特工业大学);German Research Center for AI (DFKI)(德国人工智能研究中心);Centre for Cognitive Science, TU Darmstadt(达姆施塔特工业大学认知科学中心)
InvariantCloud: A Globally Invariant, Uniquely Indexed Point Cloud Framework for Robust 6-DoF Tactile Pose Tracking
Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology(香港科学与技术大学机械与航空航天工程系);Department of Mechanical Engineering, Massachusetts Institute of Technology(麻省理工学院机械工程系);Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong(香港中文大学机械与自动化工程系)
Dextrous Tactile In-Hand Manipulation Using a Modular Reinforcement Learning Architecture
DLR Institute of Robotics and Mechatronics(DLR机器人与 mechatronics研究所);Technical University of Munich(慕尼黑工业大学);Deggendorf Institute of Technology(代根多夫应用技术大学)
TranTac: Leveraging Transient Tactile Signals for Contact-Rich Robotic Manipulation
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, Shenzhen, China(计算机科学与技术学院,哈尔滨工业大学深圳校区);State Key Laboratory of Smart Farm Technologies and Systems, Harbin, China(智能农业技术与系统国家重点实验室,哈尔滨,中国)
Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Shanghai Qizhi Institute(上海期智研究院);Department of Mechanical Engineering, Tsinghua University(清华大学机械工程系);Northwestern Polytechnical University(西北工业大学);Center on Frontiers of Computing Studies, Peking University(北京大学计算与前沿研究中心);Shanghai AI Laboratory(上海人工智能实验室)