Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Innovation and Research and Development Department, BoardWare Information System Company Ltd.（创新与研发部，BoardWare信息系统有限公司）；School of Artificial Intelligence, Nanjing Agricultural University（人工智能学院，南京农业大学）；School of Computing, National University of Singapore（计算机学院，新加坡国立大学）；School of Electronic Engineering and Automation, Guilin University of Electronic Technology（电子工程与自动化学院，桂林电子科技大学）
Yaohua Liu, Rong Fu, Amir H. Gandomi, Simon Fong, Hengjun Zhang
Tactile feedback is important for contact-rich robotic manipulation, yet effective use of tactile observations remains challenging when tactile signals are image-like, hardware-dependent, and only weakly aligned with external visual observations. This study addresses this representation-learning problem by proposing a visuo-tactile contrastive learning framework for imitation-based manipulation. The method aligns external RGB observations and calibrated tactile images in a shared embedding space using a CLIP-style objective, and integrates the resulting representation into an Action Chunking Transformer (ACT) policy. A low-cost visuo-tactile gripper (LVTG) is proposed to provide a modular and durable sensing platform for reproducible data collection, supplying tactile observations that can be used by downstream manipulation algorithms. Experiments on contact-rich manipulation tasks show that tactile feedback improves the average task completion rate from 30% for a vision-only ACT baseline to 42%, and that the proposed contrastive pretraining further increases the completion rate to 54%. These results indicate that explicitly aligning visual and tactile observations provides more useful contact-aware features for downstream policy learning than directly adding tactile images without pretraining.
Guangdong University of Technology（广东工业大学）；South China University of Technology（华南理工大学）
Humans can accurately determine whether the object in hand has slipped or not by visual and tactile perception. However, it is still a challenge for robots to detect in-hand object slip through visuo-tactile fusion. To address this issue, a novel visuo-tactile fusion deep neural network is proposed to detect slip, which is a time-dependent continuous action. By using the multi-scale temporal convolution network (MS-TCN) to extract the temporal features of visual and tactile data, the slip can be detected effectively. In this paper, a 7-dregree-of-freedom (7-DoF) robot manipulator equipped with…
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics（TARS机器人实验室）；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）；CASIA（中国科学院自动化研究所）；Tsinghua University（清华大学）；Zhongguancun Academy（中关村学院）；Beihang University（北京航空航天大学）
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tsinghua University（清华大学）；School of Information, Renmin University of China（中国人民大学信息学院）；Fudan University（复旦大学）；TARS Robotics
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University（复旦大学）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身智能重点实验室）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）；Neote AI；China Unicom（中国联通）
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University（前沿计算研究中心，计算机科学学院，北京大学）；PKU-Agibot Lab, School of Computer Science, Peking University（北京大学计算机科学学院PKU-Agibot实验室）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（国家多媒体信息处理重点实验室，计算机科学学院，北京大学）
Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli（工程学院，坎帕尼亚路易吉·范维蒂利大学）；Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Padova（信息工程学院，帕多瓦大学）
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology（智能制造装备与技术国家重点实验室）；Huazhong University of Science and Technology（华中科技大学）；Wuhan Huaweike Intelligent Technology Co., Ltd.（武汉华为凯科技有限公司）
KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation
Tongji University（同济大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；University of Hong Kong（香港大学）；Shanghai Qi Zhi Institute（上海启智研究院）；Shanghai AI Lab（上海人工智能实验室）
NVIDIA（英伟达）；Georgia Institute of Technology（佐治亚理工学院）；University of Utah（犹他大学）
Using simulation to train robot manipulation policies holds the promise of an almost unlimited amount of training data, generated safely out of harm's way. One of the key challenges of using simulation, to date, has been to bridge the reality gap, so that policies trained in simulation can be deployed in the real world. We explore the reality gap in the context of learning a contextual policy for multi-fingered robotic grasping. We propose a Grasping Objects Approach for Tactile (GOAT) robotic hands, learning to overcome the reality gap problem. In our approach we use human hand motion demonst…
A*STAR Institute for Infocomm Research (IR)（A*STAR资讯通信研究院（IR））；A*STAR Institute of High Performance Computing (IHPC)（A*STAR高性能计算研究院（IHPC））
Robots are increasingly expected to manipulate objects in ever more unstructured environments where the object properties have high perceptual uncertainty from any single sensory modality. This directly impacts successful object manipulation. In this work, we propose a reinforcement learning-based motion planning framework for object manipulation which makes use of both on-the-fly multisensory feedback and a learned attention-guided deep affordance model as perceptual states. The affordance model is learned from multiple sensory modalities, including vision and touch (tactile and force/torque)…
Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation
Center on Frontiers of Computing Studies, School of Computer Science, Peking University（前沿计算研究中心，计算机科学学院，北京大学）；PKU-Agibot Lab, School of Computer Science, Peking University（PKU-Agibot实验室，计算机科学学院，北京大学）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；Ant Group（蚂蚁集团）；Institute for Artificial Intelligence, Peking University（人工智能研究院，北京大学）
Shape-Space Deformer: Unified Visuo-Tactile Representations for Robotic Manipulation of Deformable Objects
CSIRO Robotics, Data61, CSIRO（CSIRO机器人部、Data61、CSIRO）；School of Electrical Engineering and Computer Science (EECS), The University of Queensland（电气工程与计算机科学学院（EECS）、昆士兰大学）
Integrating and Evaluating Visuo-tactile Sensing with Haptic Feedback for Teleoperated Robot Manipulation
Technische Universität Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Hessian Centre for Artificial Intelligence (Hessian.AI)（黑森州人工智能中心）；Centre for Cognitive Science (CogSci)（认知科学中心）；Nagoya University（名古屋大学）
UltraTac: Integrated Ultrasound-Augmented Visuotactile Sensor for Enhanced Robotic Perception
Shenzhen Ubiquitous Data Enabling Key Lab, Shenzhen International Graduate School, Tsinghua University（深圳 ubiquitous 数据赋能重点实验室，深圳国际研究生院，清华大学）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）
TacSE3: Equivariant SE(3) Motion Estimation from Low-Texture Visuotactile Images for In-Gripper Tracking and Compensation
Hong Kong Center for Construction Robotics, The Hong Kong University of Science and Technology, Hong Kong, China（香港建设机器人中心，香港科技大学，中国香港）；The Hong Kong University of Science and Technology (Guangzhou), China（香港科技大学（广州），中国）；School of Engineering, Great Bay University, Dongguan, China（工程学院，大湾大学，中国东莞）
University of Science and Technology Beijing（北京科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
Visuotactile sensing technology is becoming more popular in tactile sensing, but the effectiveness of the existing marker detection localization methods remains to be further explored. Instead of contour-based blob detection, this paper presents a learning-based marker localization network for GelStereo visuotactile sensing called Marknet. Specifically, the Marknet presents a grid regression architecture to incorporate the distribution of the GelStereo markers. Furthermore, a marker rationality evaluator (MRE) is modelled to screen suitable prediction results. The experimental results show tha…
BMW Group AG（宝马集团）；Imperial College of Science, Technology and Medicine（帝国理工学院）；University of Crete（克里特大学）；Institute of Molecular Biology and Biotechnology, Foundation for Research and Technology-Hellas（分子生物学与生物技术研究所，研究与技术基金会-希腊）；Technical University of Munich（慕尼黑工业大学）；Eötvös Loránd University（罗兰大学）；Foundation Robotics Labs（基础机器人实验室）；Eindhoven University of Technology（埃因霍温理工大学）
SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects (Early Version)
Tuojing Intelligence（拓景智能）；Tsinghua University（清华大学）；King’s College London（伦敦国王学院）；Southeast University（东南大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Manchester（曼彻斯特大学）；Simple AI（简单人工智能公司）；Imperial College London（伦敦帝国学院）；Carnegie Mellon University（卡内基梅隆大学）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）；The University of Hong Kong（香港大学）
Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California Berkeley（加州大学伯克利分校）；Dongguk University（东国大学）；Tsinghua University（清华大学）；University of California San Diego（加州大学圣迭戈分校）
Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
Peking University（北京大学）；The Hong Kong University of Science and Technology（香港科技大学）；Nanjing University（南京大学）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）
TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；OpenDriveLab at The University of Hong Kong（香港大学OpenDriveLab）；Shanghai Jiao Tong University（上海交通大学）；East China University of Science and Technology（华东理工大学）
Carnegie Mellon University（卡内基梅隆大学）；Brigham Young University（杨百翰大学）
Knowledge of 3-D object shape is of great importance to robot manipulation tasks, but may not be readily available in unstructured environments. While vision is often occluded during robot-object interaction, high-resolution tactile sensors can give a dense local perspective of the object. However, tactile sensors have limited sensing area and the shape representation must faithfully approximate non-contact areas. In addition, a key challenge is efficiently incorporating these dense tactile measurements into a 3-D mapping framework. In this work, we propose an incremental shape mapping method …
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
ScaleLab, Shanghai Jiao Tong University（上海交通大学ScaleLab）；D-Robotics；ViTai Robotics；The University of Hong Kong（香港大学）；Nanjing University（南京大学）；Shenzhen University（深圳大学）；Wuhan University（武汉大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）
Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables…
