Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg(乌尔姆技术大学计算机科学与人工智能系);Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);TUM School of Computation, Information and Technology, Technical University of Munich(慕尼黑技术大学计算、信息与技术学院)
Seongjin Bien, Carlo Kneissl, Tobias Jülg, Frank Fundel, Thomas Ressler-Antal, Florian Walter, Björn Ommer, Gitta Kutyniok, Wolfram Burgard
Tactile sensation is essential for contact-rich manipulation tasks. It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control. An inherent limitation of tactile sensors is that readings are available only when an object is touched. This precludes their use during planning and the initial execution phase of a task. Predicting tactile information from visual information can bridge this gap. A common approach is to learn a direct mapping from camera images to the output of vision-based tactile sensors. However, the resulting model will depend strongly on the specific setup and on how well the camera can capture the area where an object is touched. In this work, we introduce FlowTouch, a novel model for view-invariant visuo-tactile prediction. Our key idea is to use an object's local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details. FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation. Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances. We further show that the resulting tactile images can be used for downstream grasp stability prediction. Our code, datasets and videos are available at https://flowtouch.github.io/
SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation
Tuojing Intelligence(拓境智能);Tsinghua University(清华大学);Southeast University(东南大学);Stevens Institute of Technology(斯蒂文斯理工学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Manchester(曼彻斯特大学);Simple AI;Imperial College London(帝国理工学院);Carnegie Mellon University(卡内基梅隆大学);Zhejiang University(浙江大学);Beihang University(北京航空航天大学);The University of Hong Kong(香港大学)
Let Robots Feel Your Touch: Visuo-Tactile Cortical Alignment for Embodied Mirror Resonance
Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Key Laboratory of Biomedical Engineering of Hainan Province, School of Biomedical Engineering, Hainan University(海南省生物医学工程重点实验室,海南大学生物医学工程学院);School of New Media Art and Design, Beihang University(北航艺术与设计学院);MoE Key Laboratory for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology(教育部生物医学光子学重点实验室,武汉光电研究所,华中科技大学)
Tac2Real: Reliable and GPU Visuotactile Simulation for Online Reinforcement Learning and Zero-Shot Real-World Deployment
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Hong Kong University of Science and Technology(香港科技大学);Algorithms of Machine Learning and Autonomous Driving Research Lab, HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute(机器学习与自动驾驶算法研究实验室,香港科技大学深圳-香港协同创新研究院);Shanghai Jiao Tong University(上海交通大学)
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
ScaleLab, Shanghai Jiao Tong University(上海交通大学ScaleLab);D-Robotics;ViTai Robotics;The University of Hong Kong(香港大学);Nanjing University(南京大学);Shenzhen University(深圳大学);Wuhan University(武汉大学);Fudan University(复旦大学);Tsinghua University(清华大学)
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics(TARS机器人实验室);National University of Singapore(新加坡国立大学);Fudan University(复旦大学);CASIA(中国科学院自动化研究所);Tsinghua University(清华大学);Zhongguancun Academy(中关村学院);Beihang University(北京航空航天大学)
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Innovation and Research and Development Department, BoardWare Information System Company Ltd.(创新与研发部,BoardWare信息系统有限公司);School of Artificial Intelligence, Nanjing Agricultural University(人工智能学院,南京农业大学);School of Computing, National University of Singapore(计算机学院,新加坡国立大学);School of Electronic Engineering and Automation, Guilin University of Electronic Technology(电子工程与自动化学院,桂林电子科技大学)
BMW Group AG(宝马集团);Imperial College of Science, Technology and Medicine(帝国理工学院);University of Crete(克里特大学);Institute of Molecular Biology and Biotechnology, Foundation for Research and Technology-Hellas(分子生物学与生物技术研究所,研究与技术基金会-希腊);Technical University of Munich(慕尼黑工业大学);Eötvös Loránd University(罗兰大学);Foundation Robotics Labs(基础机器人实验室);Eindhoven University of Technology(埃因霍温理工大学)
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tsinghua University(清华大学);School of Information, Renmin University of China(中国人民大学信息学院);Fudan University(复旦大学);TARS Robotics
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University(复旦大学);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身智能重点实验室);Hefei University of Technology(合肥工业大学);National University of Singapore(新加坡国立大学);Neote AI;China Unicom(中国联通)
FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation
Shanghai Innovation Institute(上海创新研究院);The University of Hong Kong(香港大学);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);Shanghai University(上海大学)
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
UC Berkeley(加州大学伯克利分校);TU Dresden(德累斯顿工业大学);The Centre for Tactile Internet with Human-in-the-Loop (CeTI)(人在环触觉互联网中心(CeTI));Carnegie Mellon University(卡内基梅隆大学)
Active Visuo-Tactile Interactive Robotic Perception for Accurate Object Pose Estimation in Dense Clutter
BMW Group(宝马集团);University of Glasgow(格拉斯哥大学);The Imperial College of Science, Technology and Medicine(帝国理工学院);Donders Institute for Brain and Cognition, Radboud University(拉德堡德大学唐德斯脑认知研究所)
Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control
Chair of Robotics and Systems Intelligence, MIRMI-Munich Institute of Robotics and Machine Intelligence, Technical University of Munich(慕尼黑工业大学机器人与系统智能教席,慕尼黑机器人与机器智能研究所);Centre for Tactile Internet with Human-in-the-Loop (CeTI)(以人为中心的触觉互联网中心)
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University(前沿计算研究中心,计算机科学学院,北京大学);PKU-Agibot Lab, School of Computer Science, Peking University(北京大学计算机科学学院PKU-Agibot实验室);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机科学学院,北京大学)
ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
Massachusetts Institute of Technology(麻省理工学院);University of Washington(华盛顿大学);Harvard University(哈佛大学);University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学)
Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli(工程学院,坎帕尼亚路易吉·范维蒂利大学);Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Padova(信息工程学院,帕多瓦大学)
TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
Shanghai Jiao Tong University(上海交通大学);Xense Robotics(Xense机器人公司);Sun Yat-sen University(中山大学);Oxford(牛津大学);Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);UCSD(加州大学圣地亚哥分校)
Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs
Dept. of Electronic Engineering at Sogang University(ソガン大学电子工程系);Dept. of Immersive Media and Engineering at Sungkyunkwan University(顺天大学沉浸媒体与工程系);College of Medicine, Yonsei University(延世大学医学院)