StreamGaze：流媒体视频中的目光引导时间推理与前瞻性理解
StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos
AI总结：
StreamGaze通过引入目光引导的任务评估多模态大语言模型在流媒体视频中的时间推理和前瞻性理解能力，揭示了现有模型在目光引导下的时间推理、意图建模和前瞻性预测方面的局限性。
摘要：
流媒体视频理解需要模型不仅处理依次到来的帧，还需预测用户意图以应用于增强现实（AR）眼镜等真实场景。尽管先前的流媒体基准评估了时间推理，但没有衡量多模态大语言模型（MLLMs）能否在流媒体环境中解释或利用人类目光信号。为填补这一空白，我们引入了StreamGaze，首个评估MLLMs在流媒体视频中利用目光进行时间和前瞻性推理的有效性的基准。StreamGaze引入了目光引导的过去、现在和前瞻性任务，全面评估流媒体视频理解。这些任务评估模型是否能利用实时目光信号跟踪注意力变化并仅基于过去和当前观察的帧推断用户意图。为构建StreamGaze，我们开发了一条目光-视频问答（QA）生成管道，通过固定点提取、区域特定视觉提示和扫描路径构建，将视角视频与原始目光轨迹对齐。该管道生成时空接地的QA对，反映人类感知动态。在所有StreamGaze任务中，我们观察到最先进MLLMs与人类表现之间存在显著性能差距，突显了基于目光的时间推理、意图建模和前瞻性预测的关键限制。我们进一步提供了详细的目光提示策略、推理行为和任务特定的失败模式分析，为当前局限性和未来研究方向提供了见解。所有数据和代码均公开可用，以支持继续研究基于目光的流媒体视频理解。
英文摘要：
Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. These tasks evaluate whether models can use real-time gaze signals to follow shifting attention and infer user intentions based only on past and currently observed frames. To build StreamGaze, we develop a gaze-video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support continued research in gaze-guided streaming video understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
