ViTacWorld：用于丰富接触式机器人操作的视觉-触觉世界模型扩展
ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
浏览论文内容
中文总结 AI 辅助
研究针对丰富接触式机器人操作中视觉-触觉学习扩展难的问题，提出ViTacWorld模型，利用真实和模拟数据预训练并微调，能根据机器人动作预测视觉与触觉反馈，实现动作条件策略评估，提升策略性能。
中文摘要 AI 辅助
丰富接触式机器人操作需要物理交互线索，触觉感应至关重要，但扩展视觉-触觉机器人学习困难。我们提出ViTacWorld，一个用于可扩展丰富接触式机器人操作的动作条件视觉-触觉世界模型。它利用公共真实触觉数据集和构建的模拟环境扩展视觉-触觉-动作数据，先大规模预训练，再用真实策略展开微调。给定机器人动作，它能预测视觉观察和触觉反馈，生成视觉-触觉-动作展开。实验表明它能生成有物理意义的展开，通过可扩展数据增强改善策略性能并实现动作条件策略评估。
英文摘要
Contact-rich robot manipulation requires physical interaction cues that are often invisible to cameras, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning remains difficult because real tactile interaction data are expensive to collect, hardware-dependent, and limited in task and scene diversity. We present ViTacWorld, an action-conditioned visuo-tactile world model for scalable contact-rich robot manipulation. ViTacWorld leverages public real tactile datasets and a constructed simulation environment to scale visuo-tactile-action data, exploiting the fact that tactile signals are directly grounded in physical contact and can exhibit a smaller simulation-to-real gap than purely visual observations. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, and then finetuned with real-world policy rollouts to better match downstream manipulation behaviors. Given robot actions, ViTacWorld predicts temporally aligned visual observations and tactile feedback, enabling visuo-tactile-action rollout generation. To the best of our knowledge, ViTacWorld is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation. It serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. Project page: https://vitacworld.github.io/
发表机构
- ShanghaiTech University（上海科技大学）
- InstAdapt
机构由 AI 辅助整理，请以论文原文为准。
