VidEgoThink：评估具身AI的第一人称视角视频理解能力
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
AI总结：
本文提出VidEgoThink基准，通过视频问答、层次规划、视觉定位和奖励建模四项任务评估具身AI第一人称视角视频理解能力，实验表明现有MLLMs在此类任务上表现不佳，需进一步研究。
摘要：
多模态大语言模型（MLLMs）的最新进展为具身AI（Embodied AI）的应用开辟了新途径。基于先前的工作EgoThink，我们引入了VidEgoThink，这是一个用于评估第一人称视角视频理解能力的综合基准。为了弥合具身AI中MLLMs与低层控制之间的差距，我们设计了四个相互关联的关键任务：视频问答、层次规划、视觉定位和奖励建模。为了最小化人工标注成本，我们基于Ego4D数据集开发了一个自动数据生成流水线，利用GPT-4o的先验知识和多模态能力。随后，三名人工标注者对生成的数据进行筛选，以确保多样性和质量，最终构建了VidEgoThink基准。我们对三类模型进行了广泛实验：基于API的MLLMs、基于图像的开源MLLMs和基于视频的开源MLLMs。实验结果表明，包括GPT-4o在内的所有MLLMs在与第一人称视角视频理解相关的所有任务上表现均不佳。这些发现表明，基础模型仍需取得重大进展才能有效应用于具身AI的第一人称场景。总之，VidEgoThink反映了采用MLLMs处理第一人称视觉的研究趋势，类似于人类能力，使其能够在复杂的真实世界环境中进行主动观察和交互。
英文摘要：
Recent advancements in Multi-modal Large Language Models (MLLMs) have opened new avenues for applications in Embodied AI. Building on previous work, EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating egocentric video understanding capabilities. To bridge the gap between MLLMs and low-level control in Embodied AI, we design four key interrelated tasks: video question-answering, hierarchy planning, visual grounding and reward modeling. To minimize manual annotation costs, we develop an automatic data generation pipeline based on the Ego4D dataset, leveraging the prior knowledge and multimodal capabilities of GPT-4o. Three human annotators then filter the generated data to ensure diversity and quality, resulting in the VidEgoThink benchmark. We conduct extensive experiments with three types of models: API-based MLLMs, open-source image-based MLLMs, and open-source video-based MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform poorly across all tasks related to egocentric video understanding. These findings suggest that foundation models still require significant advancements to be effectively applied to first-person scenarios in Embodied AI. In conclusion, VidEgoThink reflects a research trend towards employing MLLMs for egocentric vision, akin to human capabilities, enabling active observation and interaction in the complex real-world environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
