EgoThink：评估视觉语言模型的第一人称视角思维能力
EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models
AI总结：
本文提出第一人称视角视觉问答基准EgoThink，涵盖六项核心能力与十二个维度，并通过评估十八个VLM揭示现有模型在第一人称思维任务上仍有较大提升空间。
摘要：
视觉语言模型（VLMs）近期在传统下游任务中展现出令人鼓舞的结果。评估研究随之兴起以衡量其能力，但大多数评估聚焦于第三人称视角，只有少数研究从第一人称视角处理特定任务。然而，VLMs从第一人称视角进行“思考”的能力——这一对推进自主智能体和机器人技术至关重要的属性——在很大程度上仍未被探索。为弥合这一研究空白，我们提出EgoThink，一个新颖的视觉问答基准，涵盖六项核心能力与十二个详细维度。该基准使用选自第一人称视频的片段构建，并配有包含第一人称信息的人工标注问答对。为全面评估VLMs，我们在EgoThink上评估了十八个流行的VLM。此外，鉴于答案的开放式格式，我们使用GPT-4作为自动评判器来计算单答案评分。实验结果表明，尽管GPT-4V在众多维度上领先，但所有被评估的VLM在第一人称视角任务中仍有相当大的提升空间。同时，增加可训练参数数量对模型在EgoThink上的性能影响最为显著。总之，EgoThink是对现有VLM评估基准的有价值补充，为具身人工智能和机器人领域的未来研究提供了不可或缺的资源。
英文摘要：
Vision-language models (VLMs) have recently shown promising results in traditional downstream tasks. Evaluation studies have emerged to assess their abilities, with the majority focusing on the third-person perspective, and only a few addressing specific tasks from the first-person perspective. However, the capability of VLMs to "think" from a first-person perspective, a crucial attribute for advancing autonomous agents and robotics, remains largely unexplored. To bridge this research gap, we introduce EgoThink, a novel visual question-answering benchmark that encompasses six core capabilities with twelve detailed dimensions. The benchmark is constructed using selected clips from egocentric videos, with manually annotated question-answer pairs containing first-person information. To comprehensively assess VLMs, we evaluate eighteen popular VLMs on EgoThink. Moreover, given the open-ended format of the answers, we use GPT-4 as the automatic judge to compute single-answer grading. Experimental results indicate that although GPT-4V leads in numerous dimensions, all evaluated VLMs still possess considerable potential for improvement in first-person perspective tasks. Meanwhile, enlarging the number of trainable parameters has the most significant impact on model performance on EgoThink. In conclusion, EgoThink serves as a valuable addition to existing evaluation benchmarks for VLMs, providing an indispensable resource for future research in the realm of embodied artificial intelligence and robotics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
