2HandedAfforder：从人类视频中学习精确可执行的双手动觉可供性
2HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videos
AI总结：
本文提出2HandedAfforder框架，从人类视频中提取双手可供性数据并训练基于VLM的预测模型，实现精确可执行的可供性区域分割，并在机器人操作中验证其有效性。
摘要：
在与物体交互时，人类能够有效地判断物体的哪些区域适合执行预期动作，即物体的可供性区域。他们还能根据要执行的任务以及需要使用单手还是双手，考虑物体区域之间的细微差异。然而，当前基于视觉的可供性预测方法往往将问题简化为朴素的物体部件分割。在这项工作中，我们提出了一个从人类活动视频数据集中提取可供性数据的框架。我们提取的2HANDS数据集包含精确的物体可供性区域分割以及作为所执行活动叙述的可供性类别标签。该数据还考虑了双手动作，即两只手协调并与一个或多个物体交互。我们提出了一个基于VLM的可供性预测模型2HandedAfforder，在该数据集上进行训练，并展示了在各种活动的可供性区域分割中优于基线的性能。最后，我们通过在机器人操作场景中的演示表明，我们预测的可供性区域是可执行的，即可以被执行任务的智能体使用。项目网站：https://sites.google.com/view/2handedafforder
英文摘要：
When interacting with objects, humans effectively reason about which regions of objects are viable for an intended action, i.e., the affordance regions of the object. They can also account for subtle differences in object regions based on the task to be performed and whether one or two hands need to be used. However, current vision-based affordance prediction methods often reduce the problem to naive object part segmentation. In this work, we propose a framework for extracting affordance data from human activity video datasets. Our extracted 2HANDS dataset contains precise object affordance region segmentations and affordance class-labels as narrations of the activity performed. The data also accounts for bimanual actions, i.e., two hands co-ordinating and interacting with one or more objects. We present a VLM-based affordance prediction model, 2HandedAfforder, trained on the dataset and demonstrate superior performance over baselines in affordance region segmentation for various activities. Finally, we show that our predicted affordance regions are actionable, i.e., can be used by an agent performing a task, through demonstration in robotic manipulation scenarios. Project-website: https://sites.google.com/view/2handedafforder
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
