从第一人称视角视频中学习精确的可供性以用于机器人操作
Learning Precise Affordances from Egocentric Videos for Robotic Manipulation
AI总结：
本文提出一个完整的可供性学习系统，利用第一人称视频自动生成精确标注，结合几何信息与视觉基础模型提升泛化能力，并构建支持工具抓取与交接的机器人操作框架，在mIoU上超越SOTA 13.8%，179次试验中成功率达77.1%。
摘要：
可供性定义为物体提供的潜在动作，对于具身AI智能体至关重要。例如，此类知识指导智能体在切割时握住刀柄，或在安全交接时握住刀刃。尽管现有方法取得了显著进展，可供性研究仍面临三个关键挑战：数据稀缺、泛化能力差和真实世界部署。具体而言，缺乏具有精确分割图的大规模可供性数据集，现有模型难以在不同领域或新颖物体与可供性类别间泛化，且鲜有工作展示在真实场景中的可部署性。在本工作中，我们通过提出一个完整的可供性学习系统来解决这些问题，该系统（1）接收第一人称视角视频并输出精确的可供性标注而无需人工标记，（2）利用几何信息和视觉基础模型来提升泛化能力，（3）引入一个支持面向可供性的机器人操作框架，如工具抓取和机器人到人的工具交接。实验结果表明，我们的模型在mIoU上超越现有技术13.8%，该框架在179次试验中实现了77.1%的成功抓取，包括对已见、未见类别和杂乱场景的评估。项目页面：https://reagan1311.github.io/affgrasp.
英文摘要：
Affordance, defined as the potential actions that an object offers, is crucial for embodied AI agents. For example, such knowledge directs an agent to grasp a knife by the handle for cutting or by the blade for safe handover. While existing approaches have made notable progress, affordance research still faces three key challenges: data scarcity, poor generalization, and real-world deployment. Specifically, there is a lack of large-scale affordance datasets with precise segmentation maps, existing models struggle to generalize across different domains or novel object and affordance classes, and little work demonstrates deployability in real-world scenarios. In this work, we address these issues by proposing a complete affordance learning system that (1) takes in egocentric videos and outputs precise affordance annotations without human labeling, (2) leverages geometric information and vision foundation models to improve generalization, and (3) introduces a framework that facilitates affordance-oriented robotic manipulation such as tool grasping and robot-to-human tool handover. Experimental results show that our model surpasses the state-of-the-art by 13.8% in mIoU, and the framework achieves 77.1% successful grasping among 179 trials, including evaluations on seen, unseen classes, and cluttered scenes. Project page: https://reagan1311.github.io/affgrasp.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
