FeelWorld：用于分层接触预测和规划的视觉触觉世界模型
FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning
浏览论文内容
中文总结 AI 辅助
研究针对现有视觉世界模型忽视触觉状态问题，提出分层视觉触觉世界模型FeelWorld，通过联合预测视觉和触觉状态、引入注意力机制及训练方法，实现接触预测与规划，实验表明其能降低预测误差并提高规划成功率。
中文摘要 AI 辅助
人类通过想象候选动作的可能结果来规划物理交互。然而，现有的视觉世界模型主要捕捉外观动态，而忽略了控制丰富接触交互的触觉状态，可能产生视觉上看似合理但违反物理动态的想象未来。我们引入了FeelWorld，一种分层视觉触觉世界模型，它联合预测未来视觉潜在状态和三种触觉状态。FeelWorld将这些状态分层组织为接触状态、编码力相关信息的3D触觉潜在状态和滑动状态。这些状态由具有明确监督的共享潜在动力学模型联合预测。为防止自由空间运动期间无关的触觉信号降低视觉预测，我们引入了一种接触门控不对称注意力机制，在接触前保持仅视觉预测路径，并在接触期间实现联合视觉触觉动力学预测。该模型还通过自回归展开和上下文噪声注入进行进一步训练，以提高对复合误差的鲁棒性。预测的接触和滑动状态还支持接触感知CEM规划。在芯片抓取、水果抓取和USB插入实验中，FeelWorld将10步LPIPS从0.084降至0.058，并在80步自回归展开后保持比视觉基线低61%的LPIPS。FeelWorld还实现了平均81.7%的零样本规划成功率，为将触觉传感纳入世界模型提供了一种有效方法。
英文摘要
Humans plan physical interactions by imagining the possible outcomes of candidate actions. However, existing visual world models primarily capture appearance dynamics while overlooking the tactile states that govern contact-rich interactions, potentially producing imagined futures that appear visually plausible but violate physical dynamics. We introduce FeelWorld, a hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states. FeelWorld organizes these states hierarchically as contact state, a 3D tactile latent that encodes force-related information, and slip state. These states are jointly predicted by a shared latent dynamics model with explicit supervision. To prevent irrelevant tactile signals during free-space motion from degrading visual prediction, we introduce a contact-gated asymmetric attention mechanism that maintains a visual-only prediction pathway before contact and enables joint visuo-tactile dynamics prediction during contact. The model is further trained with autoregressive rollouts and context noise injection to improve robustness to compounding errors. The predicted contact and slip states also support contact-aware CEM planning. Experiments on chip grasping, fruit grasping, and USB insertion show that FeelWorld reduces 10-step LPIPS from 0.084 to 0.058 and maintains an LPIPS that is 61% lower than that of the visual baseline after an 80-step autoregressive rollout. FeelWorld also achieves an average zero-shot planning success rate of 81.7%, providing an effective approach for incorporating tactile sensing into world models.
发表机构
- Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
- Imprintx Robotics（印记机器人公司）
- Beijing Academy of Artificial Intelligence（北京人工智能研究院）
机构由 AI 辅助整理，请以论文原文为准。
