超越语言：基于手部指向的自我中心视觉指称表达定位
Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Vision
AI总结：
本文提出EgoPoint-Ground数据集，通过手部指向与语音结合的指称机制，改进视觉指称表达的定位方法，并提出SV-CoT框架提升多模态理解能力。
摘要：
传统视觉定位（VG）主要依赖文本描述来定位物体，这种范式固有地面临语言歧义问题，并常忽略现实交互中普遍存在的非语言指示性线索。在自然自我中心交互中，手部指向与语音结合构成了最直观的指称机制。为此，我们引入EgoPoint-Ground，首个大规模多模态数据集，包含超过15k个复杂场景交互样本，提供丰富的多粒度标注，包括手部目标边界框对和密集语义描述。我们建立了全面的基准测试，评估了主流多模态大语言模型（MLLMs）和最新视觉定位架构。此外，我们提出SV-CoT，一种新的基线框架，将定位重新表述为结构化推理过程，通过视觉思维链范式整合手势和语言线索。大量实验表明，SV-CoT在现有方法上实现了11.7%的绝对提升，有效缓解了语义歧义，提升了智能体理解多模态物理意图的能力。数据集和代码将公开发布。
英文摘要：
Traditional Visual Grounding (VG) predominantly relies on textual descriptions to localize objects, a paradigm that inherently struggles with linguistic ambiguity and often ignores non-verbal deictic cues prevalent in real-world interactions. In natural egocentric engagements, hand-pointing combined with speech forms the most intuitive referring mechanism. To bridge this gap, we introduce EgoPoint-Ground, the first large-scale multimodal dataset dedicated to egocentric deictic visual grounding. Comprising over \textbf{15k} interactive samples in complex scenes, the dataset provides rich, multi-grained annotations including hand-target bounding box pairs and dense semantic captions. We establish a comprehensive benchmark for hand-pointing referring expression resolution, evaluating a wide spectrum of mainstream Multimodal Large Language Models (MLLMs) and state-of-the-art VG architectures. Furthermore, we propose SV-CoT, a novel baseline framework that reformulates grounding as a structured inference process, synergizing gestural and linguistic cues through a Visual Chain-of-Thought paradigm. Extensive experiments demonstrate that SV-CoT achieves an $\textbf{11.7\%}$ absolute improvement over existing methods, effectively mitigating semantic ambiguity and advancing the capability of agents to comprehend multimodal physical intents. The dataset and code will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
