Recently, world-action models (WAMs) have emerged as a powerful framework for robot manipulation. By jointly modeling action generation and future observation prediction, WAMs can act and anticipate the outcomes of the actions. However, existing WAMs typically rely on frozen pretrained encoders trained with reconstruction, which may not produce representations well suited for robot control. In this paper, we tackle the problem of learning physicallygrounded, control-relevant representations for WAMs. We propose MDF++, a visuo-tactile world-action model that learns latent representations of multisensory observations end-to-end with a diffusion backbone. While naive joint training can lead to feature collapse, MDF++ introduces a multi-task learning strategy that leverages cross-modal correlations to learn task-relevant representations. We evaluate MDF++ on a suite of contact-rich manipulation tasks, including realworld bolt screwing and nut mating, and show that it outperforms WAMs built on frozen pretrained representations by up to 50%. Additional videos are available on our website: jointmdf.github.io.
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics(TARS机器人实验室);National University of Singapore(新加坡国立大学);Fudan University(复旦大学);CASIA(中国科学院自动化研究所);Tsinghua University(清华大学);Zhongguancun Academy(中关村学院);Beihang University(北京航空航天大学)
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology(广东智能科学与技术研究院);Innovation and Research and Development Department, BoardWare Information System Company Ltd.(创新与研发部,BoardWare信息系统有限公司);School of Artificial Intelligence, Nanjing Agricultural University(人工智能学院,南京农业大学);School of Computing, National University of Singapore(计算机学院,新加坡国立大学);School of Electronic Engineering and Automation, Guilin University of Electronic Technology(电子工程与自动化学院,桂林电子科技大学)
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tsinghua University(清华大学);School of Information, Renmin University of China(中国人民大学信息学院);Fudan University(复旦大学);TARS Robotics
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University(复旦大学);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身智能重点实验室);Hefei University of Technology(合肥工业大学);National University of Singapore(新加坡国立大学);Neote AI;China Unicom(中国联通)
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University(前沿计算研究中心,计算机科学学院,北京大学);PKU-Agibot Lab, School of Computer Science, Peking University(北京大学计算机科学学院PKU-Agibot实验室);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机科学学院,北京大学)
Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli(工程学院,坎帕尼亚路易吉·范维蒂利大学);Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Padova(信息工程学院,帕多瓦大学)
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology(智能制造装备与技术国家重点实验室);Huazhong University of Science and Technology(华中科技大学);Wuhan Huaweike Intelligent Technology Co., Ltd.(武汉华为凯科技有限公司)
KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation
Tongji University(同济大学);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);University of Hong Kong(香港大学);Shanghai Qi Zhi Institute(上海启智研究院);Shanghai AI Lab(上海人工智能实验室)
Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation
Center on Frontiers of Computing Studies, School of Computer Science, Peking University(前沿计算研究中心,计算机科学学院,北京大学);PKU-Agibot Lab, School of Computer Science, Peking University(PKU-Agibot实验室,计算机科学学院,北京大学);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学);Ant Group(蚂蚁集团);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)
Shape-Space Deformer: Unified Visuo-Tactile Representations for Robotic Manipulation of Deformable Objects
CSIRO Robotics, Data61, CSIRO(CSIRO机器人部、Data61、CSIRO);School of Electrical Engineering and Computer Science (EECS), The University of Queensland(电气工程与计算机科学学院(EECS)、昆士兰大学)
Integrating and Evaluating Visuo-tactile Sensing with Haptic Feedback for Teleoperated Robot Manipulation
Technische Universität Darmstadt(达姆施塔特工业大学);German Research Center for AI (DFKI)(德国人工智能研究中心);Hessian Centre for Artificial Intelligence (Hessian.AI)(黑森州人工智能中心);Centre for Cognitive Science (CogSci)(认知科学中心);Nagoya University(名古屋大学)
UltraTac: Integrated Ultrasound-Augmented Visuotactile Sensor for Enhanced Robotic Perception
Shenzhen Ubiquitous Data Enabling Key Lab, Shenzhen International Graduate School, Tsinghua University(深圳 ubiquitous 数据赋能重点实验室,深圳国际研究生院,清华大学);RISC-V International Open Source Laboratory(RISC-V国际开源实验室)
TacSE3: Equivariant SE(3) Motion Estimation from Low-Texture Visuotactile Images for In-Gripper Tracking and Compensation
Hong Kong Center for Construction Robotics, The Hong Kong University of Science and Technology, Hong Kong, China(香港建设机器人中心,香港科技大学,中国香港);The Hong Kong University of Science and Technology (Guangzhou), China(香港科技大学(广州),中国);School of Engineering, Great Bay University, Dongguan, China(工程学院,大湾大学,中国东莞)
BMW Group AG(宝马集团);Imperial College of Science, Technology and Medicine(帝国理工学院);University of Crete(克里特大学);Institute of Molecular Biology and Biotechnology, Foundation for Research and Technology-Hellas(分子生物学与生物技术研究所,研究与技术基金会-希腊);Technical University of Munich(慕尼黑工业大学);Eötvös Loránd University(罗兰大学);Foundation Robotics Labs(基础机器人实验室);Eindhoven University of Technology(埃因霍温理工大学)
SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects (Early Version)
Tuojing Intelligence(拓景智能);Tsinghua University(清华大学);King’s College London(伦敦国王学院);Southeast University(东南大学);Stevens Institute of Technology(史蒂文斯理工学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Manchester(曼彻斯特大学);Simple AI(简单人工智能公司);Imperial College London(伦敦帝国学院);Carnegie Mellon University(卡内基梅隆大学);Zhejiang University(浙江大学);Beihang University(北京航空航天大学);The University of Hong Kong(香港大学)
Real-Time Marker Localization Learning for GelStereo Tactile Sensing
University of Science and Technology Beijing(北京科技大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Center for Excellence in Brain Science and Intelligence Technology Chinese Academy of Sciences(中国科学院脑科学与智能技术卓越创新中心)
Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California Berkeley(加州大学伯克利分校);Dongguk University(东国大学);Tsinghua University(清华大学);University of California San Diego(加州大学圣迭戈分校)
Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulation
Peking University(北京大学);The Hong Kong University of Science and Technology(香港科技大学);Nanjing University(南京大学);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机学院多媒体信息处理国家重点实验室)
TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);OpenDriveLab at The University of Hong Kong(香港大学OpenDriveLab);Shanghai Jiao Tong University(上海交通大学);East China University of Science and Technology(华东理工大学)