ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
Massachusetts Institute of Technology(麻省理工学院);University of Washington(华盛顿大学);Harvard University(哈佛大学);University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学)
Yuxiang Ma, Weihan Xu, Pengfei Ye, Anna Mei, Yuxin Ray Song, Tingle Li, Haonan Chen, Edward H. Adelson
Abstract
Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot.
We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations.
On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.
Let Robots Feel Your Touch: Visuo-Tactile Cortical Alignment for Embodied Mirror Resonance
Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Key Laboratory of Biomedical Engineering of Hainan Province, School of Biomedical Engineering, Hainan University(海南省生物医学工程重点实验室,海南大学生物医学工程学院);School of New Media Art and Design, Beihang University(北航艺术与设计学院);MoE Key Laboratory for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology(教育部生物医学光子学重点实验室,武汉光电研究所,华中科技大学)
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
ScaleLab, Shanghai Jiao Tong University(上海交通大学ScaleLab);D-Robotics;ViTai Robotics;The University of Hong Kong(香港大学);Nanjing University(南京大学);Shenzhen University(深圳大学);Wuhan University(武汉大学);Fudan University(复旦大学);Tsinghua University(清华大学)
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics(TARS机器人实验室);National University of Singapore(新加坡国立大学);Fudan University(复旦大学);CASIA(中国科学院自动化研究所);Tsinghua University(清华大学);Zhongguancun Academy(中关村学院);Beihang University(北京航空航天大学)
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Innovation and Research and Development Department, BoardWare Information System Company Ltd.(创新与研发部,BoardWare信息系统有限公司);School of Artificial Intelligence, Nanjing Agricultural University(人工智能学院,南京农业大学);School of Computing, National University of Singapore(计算机学院,新加坡国立大学);School of Electronic Engineering and Automation, Guilin University of Electronic Technology(电子工程与自动化学院,桂林电子科技大学)
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tsinghua University(清华大学);School of Information, Renmin University of China(中国人民大学信息学院);Fudan University(复旦大学);TARS Robotics
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University(复旦大学);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身智能重点实验室);Hefei University of Technology(合肥工业大学);National University of Singapore(新加坡国立大学);Neote AI;China Unicom(中国联通)
FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation
Shanghai Innovation Institute(上海创新研究院);The University of Hong Kong(香港大学);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);Shanghai University(上海大学)
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
UC Berkeley(加州大学伯克利分校);TU Dresden(德累斯顿工业大学);The Centre for Tactile Internet with Human-in-the-Loop (CeTI)(人在环触觉互联网中心(CeTI));Carnegie Mellon University(卡内基梅隆大学)
Active Visuo-Tactile Interactive Robotic Perception for Accurate Object Pose Estimation in Dense Clutter
BMW Group(宝马集团);University of Glasgow(格拉斯哥大学);The Imperial College of Science, Technology and Medicine(帝国理工学院);Donders Institute for Brain and Cognition, Radboud University(拉德堡德大学唐德斯脑认知研究所)
Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control
Chair of Robotics and Systems Intelligence, MIRMI-Munich Institute of Robotics and Machine Intelligence, Technical University of Munich(慕尼黑工业大学机器人与系统智能教席,慕尼黑机器人与机器智能研究所);Centre for Tactile Internet with Human-in-the-Loop (CeTI)(以人为中心的触觉互联网中心)
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University(前沿计算研究中心,计算机科学学院,北京大学);PKU-Agibot Lab, School of Computer Science, Peking University(北京大学计算机科学学院PKU-Agibot实验室);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机科学学院,北京大学)
Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli(工程学院,坎帕尼亚路易吉·范维蒂利大学);Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Padova(信息工程学院,帕多瓦大学)
TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
Shanghai Jiao Tong University(上海交通大学);Xense Robotics(Xense机器人公司);Sun Yat-sen University(中山大学);Oxford(牛津大学);Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);UCSD(加州大学圣地亚哥分校)
Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs
Dept. of Electronic Engineering at Sogang University(ソガン大学电子工程系);Dept. of Immersive Media and Engineering at Sungkyunkwan University(顺天大学沉浸媒体与工程系);College of Medicine, Yonsei University(延世大学医学院)
Active Visuo-Tactile Point Cloud Registration for Accurate Pose Estimation of Objects in an Unknown Workspace
BMW Group(宝马集团);University of Glasgow(格拉斯哥大学);Technische Universität München(慕尼黑工业大学);Donders Institute for Brain and Cognition(唐德斯脑与认知研究所);Radboud University(拉德堡德大学)
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology(智能制造装备与技术国家重点实验室);Huazhong University of Science and Technology(华中科技大学);Wuhan Huaweike Intelligent Technology Co., Ltd.(武汉华为凯科技有限公司)
SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation
Tuojing Intelligence(拓境智能);Tsinghua University(清华大学);Southeast University(东南大学);Stevens Institute of Technology(斯蒂文斯理工学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Manchester(曼彻斯特大学);Simple AI;Imperial College London(帝国理工学院);Carnegie Mellon University(卡内基梅隆大学);Zhejiang University(浙江大学);Beihang University(北京航空航天大学);The University of Hong Kong(香港大学)
KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation
Tongji University(同济大学);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);University of Hong Kong(香港大学);Shanghai Qi Zhi Institute(上海启智研究院);Shanghai AI Lab(上海人工智能实验室)
CAAT: Contact-Aware Attention Scaling and Tactile Masking for Data-Efficient Contact-Rich Manipulation
ShanghaiTech University(上海科技大学);Beihang University(北京航空航天大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Zhejiang University(浙江大学);BUPT(北京邮电大学)