在MLLM的眼中：基于凝视引导提示的自我中心视频意图理解基准测试
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
AI总结：
针对现有基准忽视凝视在自我中心视频意图理解中的作用，提出EgoGazeVQA基准及凝视引导提示方法，整合空间、时间与意图线索，显著提升MLLMs对用户意图的解读性能。
摘要：
先进多模态大语言模型（MLLMs）的出现显著增强了AI助手跨模态处理复杂信息的能力。近期，自我中心视频通过统一坐标系直接捕捉用户注视、动作和上下文，为利用MLLMs实现主动且个性化的AI用户体验提供了令人兴奋的机会。然而，现有基准测试忽视了凝视作为用户意图指标的关键作用。为弥补这一空白，我们引入了EgoGazeVQA，一个基于凝视引导的自我中心视频问答基准测试，利用凝视信息来提升对较长日常视频的理解。EgoGazeVQA包含由MLLMs生成并经人工标注者精炼的基于凝视的问答对。我们的实验表明，现有MLLMs难以准确解读用户意图。相比之下，我们提出的凝视引导意图提示方法通过整合空间、时间及意图相关线索，显著提升了性能。我们进一步进行了凝视相关微调实验，并分析了凝视估计准确性对提示效果的影响。这些结果强调了凝视在自我中心场景中实现更个性化和高效AI助手的价值。项目页面：https://taiyi98.github.io/projects/EgoGazeVQA
英文摘要：
The emergence of advanced multimodal large language models (MLLMs) has significantly enhanced AI assistants' ability to process complex information across modalities. Recently, egocentric videos, by directly capturing user focus, actions, and context in an unified coordinate, offer an exciting opportunity to enable proactive and personalized AI user experiences with MLLMs. However, existing benchmarks overlook the crucial role of gaze as an indicator of user intent. To address this gap, we introduce EgoGazeVQA, an egocentric gaze-guided video question answering benchmark that leverages gaze information to improve the understanding of longer daily-life videos. EgoGazeVQA consists of gaze-based QA pairs generated by MLLMs and refined by human annotators. Our experiments reveal that existing MLLMs struggle to accurately interpret user intentions. In contrast, our gaze-guided intent prompting methods significantly enhance performance by integrating spatial, temporal, and intent-related cues. We further conduct experiments on gaze-related fine-tuning and analyze how gaze estimation accuracy impacts prompting effectiveness. These results underscore the value of gaze for more personalized and effective AI assistants in egocentric settings. Project page: https://taiyi98.github.io/projects/EgoGazeVQA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
