arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.08255cs.ROcs.LG
收藏

FlowTouch: 视点不变的视觉-触觉预测

FlowTouch: View-Invariant Visuo-Tactile Prediction

Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg(乌尔姆技术大学计算机科学与人工智能系);Ludwig Maximilian University of Munich(慕尼黑路德维希-马克西米利安大学);TUM School of Computation, Information and Technology, Technical University of Munich(慕尼黑技术大学计算、信息与技术学院)

Seongjin Bien, Carlo Kneissl, Tobias Jülg, Frank Fundel, Thomas Ressler-Antal, Florian Walter, Björn Ommer, Gitta Kutyniok, Wolfram Burgard

AI总结:

FlowTouch通过利用物体的局部3D网格实现视点不变的视觉-触觉预测,有效弥补仿真与现实之间的差距,并提升抓取稳定性预测能力。

摘要:

触觉感知对于涉及频繁接触的操作任务至关重要。它提供直接的反馈,关于物体几何形状、表面属性和相互作用力,从而增强感知能力并实现精细控制。触觉传感器的一个固有局限性是,只有在物体被触摸时才能获得读数。这限制了它们在任务规划和初始执行阶段的应用。通过视觉信息预测触觉信息可以弥补这一差距。一种常见的方法是学习从相机图像到基于视觉的触觉传感器输出的直接映射。然而,所得到的模型强烈依赖于特定的设置以及相机捕捉被触摸区域的能力。在本工作中,我们介绍了FlowTouch,一种新的视点不变的视觉-触觉预测模型。我们的核心思想是使用物体的局部3D网格来编码丰富信息以预测触觉模式,同时抽象掉场景依赖的细节。FlowTouch集成了场景重建和基于流匹配的模型用于图像生成。我们的结果表明,FlowTouch能够弥合仿真到现实的差距,并泛化到新的传感器实例。我们进一步展示了所得到的触觉图像可用于下游的抓取稳定性预测。我们的代码、数据集和视频可在https://flowtouch.github.io/上获得。

英文摘要:

Tactile sensation is essential for contact-rich manipulation tasks. It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control. An inherent limitation of tactile sensors is that readings are available only when an object is touched. This precludes their use during planning and the initial execution phase of a task. Predicting tactile information from visual information can bridge this gap. A common approach is to learn a direct mapping from camera images to the output of vision-based tactile sensors. However, the resulting model will depend strongly on the specific setup and on how well the camera can capture the area where an object is touched. In this work, we introduce FlowTouch, a novel model for view-invariant visuo-tactile prediction. Our key idea is to use an object's local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details. FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation. Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances. We further show that the resulting tactile images can be used for downstream grasp stability prediction. Our code, datasets and videos are available at https://flowtouch.github.io/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑