基于大型多模态模型的以自我为中心的面向指令的可用性预测
Egocentric Instruction-oriented Affordance Prediction via Large Multimodal Model
AI总结：
针对现有工作忽视可用性依赖任务/指令的问题，本文提出含一万五千个三元组的自我中心视角数据集，通过“针对验证器的搜索”流水线让LMM逐步预测并自验证可用性，解锁新预测能力且性能出色。
摘要：
可用性在智能机器人的物体操作场景中至关重要。本文认为，可用性应依赖于任务/指令，这一点被许多先前的工作所忽视。也就是说，即使对于同一物体，不同的指令也会导致不同的操作区域和方向。基于这一观察，我们提出了一个新的数据集，包含一万五千个物体-指令-可用性三元组。数据集中的所有场景均来自自我中心视角，旨在模拟类人机器人的视角。此外，我们通过实现“针对验证器的搜索”流水线，研究了如何让大型多模态模型（LMMs）作为可用性预测器。我们要求LMM逐步预测可用性，在迭代过程中每一步的输出都由其自身验证，以模仿推理过程。实验表明，我们的方法不仅解锁了新的面向指令的可用性预测能力，还在整体上取得了出色的性能。
英文摘要：
Affordance is crucial for intelligent robots in the context of object manipulation. In this paper, we argue that affordance should be task-/instruction-dependent, which is overlooked by many previous works. That is, different instructions can lead to different manipulation regions and directions even for the same object. According to this observation, we present a new dataset comprising fifteen thousand object-instruction-affordance triplets. All scenes in the dataset are from an egocentric viewpoint, designed to approximate the perspective of a human-like robot. Furthermore, we investigate how to enable large multimodal models (LMMs) to serve as affordance predictors by implementing a ``search against verifiers'' pipeline. An LMM is asked to progressively predict affordances, with the output at each step being verified by itself during the iterative process, imitating a reasoning process. Experiments show that our method not only unlocks new instruction-oriented affordance prediction capabilities, but also achieves outstanding performance broadly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
