你看到我指向的是什么？基于手势的 egocentric 视频问答
Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering
AI总结：
本文提出 EgoPointVQA 数据集和基准，结合 Hand Intent Tokens 提升手势引导的 egocentric 问答性能，HINT-14B 在多个任务上超越现有最佳模型。
摘要：
理解并回答基于用户指向手势的问题对下一代 egocentric AI 助手至关重要。然而，当前多模态大语言模型 (MLLMs) 在此类任务上表现不佳，因为缺乏丰富手势数据以及其对 egocentric 视频中细粒度指向意图推断能力有限。为此，我们引入 EgoPointVQA，一个用于手势引导的 egocentric 问答的基准数据集，包含 4000 个合成和 400 个真实世界视频，涵盖多种指涉推理任务。基于此，我们进一步提出 Hand Intent Tokens (HINT)，通过使用现成的重建模型从 3D 手部关键点生成 tokens，并将其与模型输入交织，以提供明确的空间和时间上下文以解释指向意图。我们展示了我们的模型在不同 backbones 和模型大小上均优于其他方法。特别是 HINT-14B 在平均 6 个任务上达到 68.1% 准确率，超越 state-of-the-art 模型 InternVL3-14B 6.6%。为促进开放研究，我们将发布代码、模型和数据集。项目页面：https://yuuraa.github.io/papers/choi2026egovqa
英文摘要：
Understanding and answering questions based on a user's pointing gesture is essential for next-generation egocentric AI assistants. However, current Multimodal Large Language Models (MLLMs) struggle with such tasks due to the lack of gesture-rich data and their limited ability to infer fine-grained pointing intent from egocentric video. To address this, we introduce EgoPointVQA, a dataset and benchmark for gesture-grounded egocentric question answering, comprising 4000 synthetic and 400 real-world videos across multiple deictic reasoning tasks. Built upon it, we further propose Hand Intent Tokens (HINT), which encodes tokens derived from 3D hand keypoints using an off-the-shelf reconstruction model and interleaves them with the model input to provide explicit spatial and temporal context for interpreting pointing intent. We show that our model outperforms others in different backbones and model sizes. In particular, HINT-14B achieves 68.1% accuracy, on average over 6 tasks, surpassing the state-of-the-art, InternVL3-14B, by 6.6%. To further facilitate the open research, we will release the code, model, and dataset. Project page: https://yuuraa.github.io/papers/choi2026egovqa
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
