arXiv 2402.112652024-05-24cs.CV

超越字面描述：理解并定位与人类意图对齐的开放世界物体

Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions

Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；University of Science and Technology Beijing (USTB)（北京科技大学）

Wenxuan Wang, Yisi Zhang, Xingjian He, Yichen Yan, Zijia Zhao, Xinlong Wang, Jing Liu

备注：: This work has been accepted by ACL 2024 (Findings)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出意图驱动视觉定位（IVG）任务及IntentionVG数据集，旨在理解基于意图的表达并定位开放世界物体，通过多场景和以自我为中心视角的实验验证了方法的有效性。

摘要：

视觉定位（VG）旨在定位与给定自然语言表达相匹配的前景实体。以往经典VG任务的数据集和方法主要依赖于一个先验假设，即给定的表达必须字面地指代目标物体，这极大地阻碍了智能体在现实场景中的实际部署。由于用户通常倾向于为期望的物体提供基于意图的表达，而非涵盖所有细节，因此智能体有必要解读意图驱动的指令。为此，在本工作中，我们进一步推进到意图驱动的视觉-语言（V-L）理解。为了将经典VG推向人类意图解读，我们提出了一个新的意图驱动视觉定位（IVG）任务，并构建了一个名为IntentionVG的大规模IVG数据集，包含自由形式的意图表达。考虑到实际智能体需要移动并在各种场景中寻找特定目标以实现定位任务，我们的IVG任务和IntentionVG数据集都考虑了多场景感知和以自我为中心视角的关键特性。此外，我们设置了多种类型的模型作为基线以实现我们的IVG任务。在我们IntentionVG数据集和基线上的大量实验证明了我们方法在V-L领域的必要性和有效性。为了促进该方向的未来研究，我们新构建的数据集和基线将在https://github.com/Rubics-Xuan/IVG公开提供。

英文摘要：

Visual grounding (VG) aims at locating the foreground entities that match the given natural language expressions. Previous datasets and methods for classic VG task mainly rely on the prior assumption that the given expression must literally refer to the target object, which greatly impedes the practical deployment of agents in real-world scenarios. Since users usually prefer to provide intention-based expression for the desired object instead of covering all the details, it is necessary for the agents to interpret the intention-driven instructions. Thus, in this work, we take a step further to the intention-driven visual-language (V-L) understanding. To promote classic VG towards human intention interpretation, we propose a new intention-driven visual grounding (IVG) task and build a large-scale IVG dataset termed IntentionVG with free-form intention expressions. Considering that practical agents need to move and find specific targets among various scenarios to realize the grounding task, our IVG task and IntentionVG dataset have taken the crucial properties of both multi-scenario perception and egocentric view into consideration. Besides, various types of models are set up as the baselines to realize our IVG task. Extensive experiments on our IntentionVG dataset and baselines demonstrate the necessity and efficacy of our method for the V-L field. To foster future research in this direction, our newly built dataset and baselines will be publicly available at https://github.com/Rubics-Xuan/IVG.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。