发表机构
Center for Humanoid Research, KIST; Korea University（韩国科学技术院人形机器人研究中心; 韩国大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究如何利用多手指手上的稀疏触觉信号补充预训练视觉姿态跟踪器以应对遮挡，提出运动感知手指级编码器，经多级别评估对比，验证其能提升跟踪成功率，在下游任务表现更好并提供真实世界演示。
AI 中文摘要
灵巧的手中操作需要连续的6D姿态跟踪，但操作手指不可避免地会遮挡相机视野中的物体。我们研究如何构建多手指手上已有的稀疏触觉信号，包括本体感觉、近端力/扭矩和二元接触，以在遮挡情况下补充预训练的视觉姿态跟踪器。我们提出了一种运动感知手指级编码器，并通过三个评估级别将其与四种替代设计进行系统比较：逐帧细化、顺序开环跟踪和闭环操作。我们的实验表明：（i）逐帧评估无法区分编码器质量，而顺序跟踪会将架构差异放大多达15倍；（ii）结构化编码器学习特定任务的跨模态门控，在没有明确监督的情况下，仅使用视觉进行平移，并将一个注意力头用于触觉进行旋转；（iii）具有4个令牌的紧凑手指级令牌化优于扁平融合和关节级表示，后者通过范数优势抑制视觉。我们验证了改进的跟踪在下游重新定向任务中具有更高的成功率，并提供了定性的真实世界演示。我们的项目页面可在这个https URL上获取。
英文摘要
Dexterous in-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera. We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact, to complement a pretrained visual pose tracker under occlusion. We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation: per-frame refinement, sequential open-loop tracking, and closed-loop manipulation. Our experiments reveal that (i) per-frame evaluation cannot distinguish encoder quality, while sequential tracking amplifies architectural differences by up to 15 times; (ii) the structured encoder learns task-specific cross-modal gating, using vision exclusively for translation and dedicating one attention head to haptics for rotation, without explicit supervision; and (iii) compact finger-level tokenization with 4 tokens outperforms both flat fusion and joint-level representations, which suppress vision through norm dominance. We validate that improved tracking yields higher success in a downstream reorientation task and provide qualitative real-world demonstrations. Our project page is available at https://cold-young.github.io/kine-fuse/.
Comments8 pages, 10 figures. Accepted for presentation at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026)
