Eyes on VLM: 视觉语言模型中注视跟随与社会性注视预测的基准测试
Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models
AI总结：
提出EyeVLM评估框架，通过零样本和微调方式测试视觉语言模型在注视跟随（几何视觉处理）和社会性注视预测（社交推理）两个核心任务上的能力，发现当前VLM缺乏精确的注视理解能力。
摘要：
视觉语言模型（VLM）已迅速演变为具有强大零样本泛化能力的通用多模态推理器。在此背景下，VLM 可极大促进人类注视与注意力的分析——这是人类行为理解的核心任务，需要推理物理场景以及活动、交互和社会背景。然而，VLM 能在多大程度上可靠地理解人类注视及相关注意行为仍基本未被探索。本文提出 EyeVLM，一个跨两个互补维度（任务和模型）对 VLM 注视理解能力进行系统评估的框架。为评估注视理解能力，我们聚焦两个核心任务。第一个是注视跟随，即预测一个人注视的二维位置，侧重于几何和视觉处理，需要精确理解人脸、注意力方向、3D 场景结构以及被注视目标的空间定位。第二个是社会性注视预测，需要对多人交互（如相互注视和共享注意）进行社会和关系推理，可能更受益于 VLM 中的 LLM 语义推理能力。在模型方面，EyeVLM 通过两种方式评估这些任务：零样本设置，使用多种最先进的开源和闭源 VLM，探索不同提示策略；以及基于任务特定问答对的微调方法，研究模型规模和数据规模的影响。作为基准，我们依赖现有注视理解数据集，并与最先进的纯视觉模型进行系统比较。总体而言，我们的结果表明，当前 VLM 缺乏精确的注视理解能力。虽然标准训练有助于缩小与视觉模型的差距，但仍需显著改进。
英文摘要：
Vision-language models (VLMs) have rapidly evolved into general-purpose multimodal reasoners with strong zero-shot generalization. In this context, VLMs could greatly benefit the analysis of human gaze and attention, a central task in human behavior understanding that requires reasoning about the physical scene as well as the activity, interactions, and social context. However, the extent to which VLMs can reliably understand human gaze and related attentional behaviors remains largely unexplored. In this work, we present EyeVLM, a systematic evaluation framework for gaze understanding in VLMs across two complementary dimensions: tasks and models. To assess gaze understanding capabilities, we focus on two core tasks. The first, gaze following, i.e., predicting the 2D location where a person is looking, has a geometric and visual processing focus, requiring a precise understanding of the human face, attention direction, 3D scene structure, and spatial grounding of attended targets. The second, social gaze prediction, requires social and relational reasoning over multi-person interactions (e.g., mutual gaze and shared attention), and may benefit more from the LLM semantic reasoning capabilities within VLMs. Regarding models, EyeVLM evaluates these tasks in two ways: a zero-shot setting with a diverse set of state-of-the-art open- and closed-source VLMs, exploring different prompting strategies; and a fine-tuning approach based on task-specific QA pairs, studying the impact of model scale and data scale. As benchmarks, we rely on existing gaze understanding datasets and perform a systematic comparison with state-of-the-art purely visual models. Overall, our results show that current VLMs lack precise gaze understanding capabilities. While standard training helps reduce the gap with visual models, significant improvements are still needed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
