FingerEye：通过连续视觉-触觉感知学习灵巧操作
FingerEye: Learning Dexterous Manipulation with Continuous Vision-Tactile Sensing
AI总结：
FingerEye通过连续视觉-触觉感知提升机器人灵巧操作，结合视觉和触觉反馈，在模拟和现实环境中使腕部策略的成功率提升超30个百分点。
摘要：
灵巧的机器人操作需要从接触前的接近到接触启动和接触后的控制保持信息丰富的感知。我们介绍了FingerEye，一种通过连续视觉-触觉反馈增强机器人灵巧性的感知和学习框架。在感知方面，FingerEye整合双目RGB相机和一个合规的接触接口，以支持接触前后的同时感知。接触前，指尖相机提供近距离视觉线索和隐式立体视觉，用于精确接近和物体定位。接触后，标记跟踪的变形提供接触 wrench 感知的代理。在学习方面，我们构建了真实和模拟的基础设施用于数据收集和评估，系统研究了多项FingerEye传感器的学习策略-接口设计，并开发了FingerEye Policy，该策略通过组结构化的模态融合来减少模态捷径并更好地利用分布式的指尖反馈。在七个接触敏感的任务设置中，FingerEye在模拟和现实世界中均使腕部策略的平均成功率提高了超过30个百分点。
英文摘要：
Dexterous robotic manipulation requires perception that remains informative from pre-contact approach to contact initiation and post-contact control. We introduce FingerEye, a sensing and learning framework that strengthens robotic dexterity through continuous vision-tactile feedback throughout interaction. On the sensing side, FingerEye integrates binocular RGB cameras with a compliant contact interface to support perception both before and after contact. Before contact, the fingertip cameras provide close-range visual cues and implicit stereo for precise approach and object localization. After contact, marker-tracked deformation of the compliant ring provides a proxy for contact wrench sensing. On the learning side, we build real-and-sim infrastructure for data collection and evaluation, systematically study policy-interface designs for learning with multiple FingerEye sensors, and develop FingerEye Policy, which applies group-structured modality fusion to reduce modality shortcuts and better exploit distributed fingertip feedback. Across seven contact-sensitive task settings, FingerEye improves wrist-only policy by over 30 percentage points in mean success rate in both simulation and the real world.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
