RGB-S: 用于鲁棒灵巧操作的图像对齐触觉显著性
Tac2Pix: Image-Space Visuo-Tactile Fusion for Dexterous Manipulation
AI总结：
提出RGB-S框架，通过正向运动学和相机标定将触觉传感器位置投影到RGB图像平面，生成力调制高斯显著性图，显式对齐触觉与视觉，在严重遮挡下灵巧操作成功率提升26.7个百分点。
摘要：
有效的视觉-触觉整合对于机器人灵巧操作至关重要，尤其是在视觉观测不可靠或被遮挡时。然而，将稀疏、异构的触觉测量与密集的视觉表示鲁棒地对齐仍然是一个基本挑战。大多数现有方法需要策略从有限的演示中隐式学习跨模态对应关系，而不利用几何先验。因此，它们在视觉观测退化时往往数据效率低且泛化能力差。为解决这一限制，我们提出一个框架，显式地将物理接触锚定在图像域中。利用机器人正向运动学和相机标定，我们将触觉传感器位置直接投影到RGB图像平面上。然后，我们渲染力调制的高斯显著性图，以模拟由运动学和标定误差引起的空间不确定性。通过零初始化的条件架构整合这些2D空间锚点，我们的方法将物理接触先验注入标准视觉骨干网络，同时保留预训练的视觉表示。我们在模拟和现实世界的六项灵巧操作任务中评估了我们的方法，在严重视觉遮挡下。现实世界实验表明，在图像域中显式的RGB-S锚定将现实世界遮挡操作成功率比最强的隐式视觉-触觉基线提高了26.7个百分点，表明其空间推理能力和对遮挡的鲁棒性得到了改善。项目页面：touch-as-saliency.github.io
英文摘要：
Vision and touch are naturally complementary for dexterous manipulation. Yet they are represented in fundamentally different forms: visual observations are dense and spatially structured, whereas tactile measurements are sparse, heterogeneous, and robot-centric. This mismatch makes it challenging to establish explicit spatial correspondence between the two modalities, particularly when they are fused in latent space. However, each tactile measurement is associated with a known location on the robot's kinematic chain, offering a localization prior that could bridge this representational gap. This raises a natural question: can this localization prior be used to construct a shared representation that explicitly aligns touch with vision? We introduce Tac2Pix, an image-space visuo-tactile fusion framework that projects sparse tactile measurements onto the RGB image plane using forward kinematics and camera calibration, then renders them as force-aware saliency maps. By expressing touch in the same pixel space as vision, Tac2Pix directly leverages pretrained 2D visual representations. We further employ zero-initialized input adaptation to preserve the original visual pathway at the start of policy learning. To assess whether this geometry-based interface yields consistent policy gains, we evaluate Tac2Pix on three simulated and three real-world dexterous manipulation tasks. In simulation, Tac2Pix consistently improves visuo-tactile policy learning across tasks, policy architectures, and visual backbones. In real-world experiments, it improves average success under visual occlusion by 26.7 percentage points over the strongest latent visuo-tactile fusion baseline, with gains persisting under both random and physical occlusions. Together, these results support image space as an effective interface for fusing sparse touch with pretrained visual representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
