多模态大语言模型理解指认吗？在自我中心视觉中的基准测试与增强指认推理
Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision
AI总结：
本文提出EgoPoint-Bench基准测试，用于评估和提升自我中心视角下的多模态指认推理能力，通过11k高质量样本展示细调合成数据能显著提升性能和现实迁移能力。
摘要：
自我中心AI代理，如智能眼镜，依赖指认手势解决自然语言命令中的指认歧义。然而，尽管多模态大语言模型（MLLMs）有所进展，当前系统往往无法精确地将指认的空问语义进行定位。相反，它们依赖于与视觉接近度或物体显著性相关的虚假相关性，我们称之为'指认幻觉'。为了解决这一差距，我们引入EgoPoint-Bench，一个全面的问题回答基准，用于评估和增强自我中心视角下的多模态指认推理。该基准包含超过11k高质量的模拟和现实世界样本，涵盖五个评估维度和三个指认复杂性级别。广泛实验表明，尽管最先进的专有和开源模型在自我中心指认上表现不佳，但经过我们在合成数据上微调的模型实现了显著的性能提升和稳健的仿真到现实泛化能力。本工作强调了空间感知监督的重要性，并提供了一条通往精确自我中心AI助手的可扩展路径。项目页面：https://guyyyug.github.io/EgoPoint-Bench/
英文摘要：
Egocentric AI agents, such as smart glasses, rely on pointing gestures to resolve referential ambiguities in natural language commands. However, despite advancements in Multimodal Large Language Models (MLLMs), current systems often fail to precisely ground the spatial semantics of pointing. Instead, they rely on spurious correlations with visual proximity or object saliency, a phenomenon we term "Referential Hallucination." To address this gap, we introduce EgoPoint-Bench, a comprehensive question-answering benchmark designed to evaluate and enhance multimodal pointing reasoning in egocentric views. Comprising over 11k high-fidelity simulated and real-world samples, the benchmark spans five evaluation dimensions and three levels of referential complexity. Extensive experiments demonstrate that while state-of-the-art proprietary and open-source models struggle with egocentric pointing, models fine-tuned on our synthetic data achieve significant performance gains and robust sim-to-real generalization. This work highlights the importance of spatially aware supervision and offers a scalable path toward precise egocentric AI assistants. Project page: https://guyyyug.github.io/EgoPoint-Bench/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
