Tactile sensors are increasingly integrated into dexterous robotic manipulators to enhance contact perception. However, learning manipulation policies that rely on tactile sensing remains challenging, primarily due to the trade-off between fidelity and computational cost of soft-body simulations. To address this, we present ETac, a tactile simulation framework that models elastomeric soft-body interactions with both high fidelity and efficiency. ETac employs a lightweight data-driven deformation propagation model to capture soft-body contact dynamics, achieving high simulation quality and boosting efficiency that enables large-scale policy training. When serving as the simulation backend, ETac produces surface deformation estimates comparable to FEM and demonstrates applicability for modeling real tactile sensors. Then, we showcase its capability in training a blind grasping policy that leverages large-area tactile feedback to manipulate diverse objects. Running on a single RTX 4090 GPU, ETac supports reinforcement learning across 4,096 parallel environments, achieving a total throughput of 869 FPS. The resulting policy reaches an average success rate of 84.45% across four object types, underscoring ETac's potential to make tactile-based skill learning both efficient and scalable.
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University(前沿计算研究中心,计算机科学学院,北京大学);PKU-Agibot Lab, School of Computer Science, Peking University(北京大学计算机科学学院PKU-Agibot实验室);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机科学学院,北京大学)
High resolution tactile sensors are often bulky and have shape profiles that make them awkward for use in manipulation. This becomes important when using such sensors as fingertips for dexterous multi-fingered hands, where boxy or planar fingertips limit the available set of smooth manipulation strategies. High resolution optical based sensors such as GelSight have until now been constrained to relatively flat geometries due to constraints on illumination geometry.Here, we show how to construct a rounded fingertip that utilizes a form of light piping for directional illumination. Our sensors c…
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology(智能制造装备与技术国家重点实验室);Huazhong University of Science and Technology(华中科技大学);Wuhan Huaweike Intelligent Technology Co., Ltd.(武汉华为凯科技有限公司)
Tactile sensing is an essential capability for robots that carry out dexterous manipulation tasks. While cameras, Lidars and other remote sensors can assess a scene globally and instantly, tactile sensors can reduce their measurement uncertainties and gain information about the local physical interactions between the in-contact objects and the robot, that are often not accessible via remote sensing. Tactile sensors can be grouped into two main categories: electronic tactile skins and camera based optical tactile sensors. The former are slim and can be fitted to different body parts, whereas th…
KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation
Tongji University(同济大学);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);University of Hong Kong(香港大学);Shanghai Qi Zhi Institute(上海启智研究院);Shanghai AI Lab(上海人工智能实验室)
University of Utah(犹他大学);NVIDIA Corporation(英伟达公司)
While tactile skins have been shown to be useful for detecting collisions between a robotic arm and its environment, they have not been extensively used for improving robotic grasping and in-hand manipulation. We propose a novel sensor design for use in covering existing multi-fingered robot hands. We analyze the performance of four different piezoresistive materials using both fabric and anti-static foam substrates in benchtop experiments. We find that although the piezoresistive foam was designed as packing material and not for use as a sensing substrate, it performs comparably with fabrics …
Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation
Shanghai Jiao Tong University(上海交通大学);State Key Laboratory of General Artificial Intelligence, BIGAI(北京通用人工智能研究院 通用人工智能全国重点实验室);Sharpa Robotics;Beijing Institute of Technology(北京理工大学)
FingerEye: Learning Dexterous Manipulation with Continuous Vision-Tactile Sensing
National University of Singapore(新加坡国立大学);RoboScience(机器人科学);Huazhong University of Science and Technology(华中科技大学);South China University of Technology(华南理工大学)
Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation
Center on Frontiers of Computing Studies, School of Computer Science, Peking University(前沿计算研究中心,计算机科学学院,北京大学);PKU-Agibot Lab, School of Computer Science, Peking University(PKU-Agibot实验室,计算机科学学院,北京大学);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学);Ant Group(蚂蚁集团);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
Data-Driven Optimization of Tactile Sensor Configurations for Efficient Dexterous Manipulation
ShanghaiTech University, School of Information Science and Technology(上海科技大学信息科学与技术学院);University of Alberta(阿尔伯塔大学);Oklahoma State University(俄克拉荷马州立大学);University of Colorado Denver, Department of Computer Science and Engineering(科罗拉多大学丹佛分校计算机科学与工程系);Department of Robotics and Mechatronics Engineering, Kennesaw State University(凯斯西储大学机器人与机电工程系)