基于自我中心视觉的文本驱动可供性学习
Text-driven Affordance Learning from Egocentric Vision
AI总结：
本文提出文本驱动的自我中心可供性学习，利用文本指令预测接触点与操作轨迹，并通过伪标注流程构建 TextAFF80K 数据集，扩展指代表达理解模型实现多种真实交互。
摘要：
视觉可供性学习是机器人理解如何与物体交互的关键组成部分。该领域的传统方法依赖于预定义的物体和动作，难以捕捉真实世界场景中的多样化交互。我们方法的核心思想是利用文本指令，面向广泛物体的各类可供性。该方法同时覆盖手—物交互和工具—物交互。我们提出文本驱动的可供性学习，旨在遵循文本指令，从自我中心视角学习接触点和操作轨迹。在我们的任务中，接触点以热图表示，操作轨迹则表示为坐标序列，其中包含用于各类操作的线性运动和旋转运动。然而，在为该任务收集数据时，对这些多样化交互进行人工标注成本高昂。为此，我们提出一种伪数据集创建流程，并构建了一个大型伪训练数据集 TextAFF80K：该数据集包含超过 8 万个由接触点、轨迹、图像和文本元组组成的实例。我们扩展了现有的指代表达理解模型以适配本任务，实验结果表明，我们的方法能够稳健地处理多种可供性，为真实世界场景中的可供性学习确立了新标准。
英文摘要：
Visual affordance learning is a key component for robots to understand how to interact with objects. Conventional approaches in this field rely on pre-defined objects and actions, falling short of capturing diverse interactions in realworld scenarios. The key idea of our approach is employing textual instruction, targeting various affordances for a wide range of objects. This approach covers both hand-object and tool-object interactions. We introduce text-driven affordance learning, aiming to learn contact points and manipulation trajectories from an egocentric view following textual instruction. In our task, contact points are represented as heatmaps, and the manipulation trajectory as sequences of coordinates that incorporate both linear and rotational movements for various manipulations. However, when we gather data for this task, manual annotations of these diverse interactions are costly. To this end, we propose a pseudo dataset creation pipeline and build a large pseudo-training dataset: TextAFF80K, consisting of over 80K instances of the contact points, trajectories, images, and text tuples. We extend existing referring expression comprehension models for our task, and experimental results show that our approach robustly handles multiple affordances, serving as a new standard for affordance learning in real-world scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
