ECBench：多模态基础模型能否理解自我中心世界？一个整体具身认知基准
ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark
AI总结：
本文提出ECBench基准与ECEval评估系统，通过30个具身认知维度、多样自我中心视频及精细标注，系统评测多类LVLMs的具身认知能力。
摘要：
大型视觉-语言模型（LVLMs）对机器人泛化能力的增强正日益明显。因此，基于自我中心视频的LVLMs具身认知能力备受关注。然而，当前用于具身视频问答的数据集缺乏全面、系统的评估框架。机器人自我认知、动态场景感知和幻觉等关键具身认知问题很少得到解决。为应对这些挑战，我们提出ECBench，这是一个高质量基准，旨在系统评估LVLMs的具身认知能力。ECBench具有多样化的场景视频来源、开放且多样的问题格式，以及30个具身认知维度。为确保质量、平衡性和高度视觉依赖性，ECBench采用与类别无关的精细人工标注和多轮问题筛选策略。此外，我们引入ECEval，这是一个确保指标公平性和合理性的综合评估系统。利用ECBench，我们对专有、开源和任务专用LVLMs进行了广泛评估。ECBench在推进LVLMs具身认知能力方面具有关键作用，为开发可靠的具身智能体核心模型奠定了坚实基础。所有数据和代码均可在https://github.com/Rh-Dang/ECBench获取。
英文摘要：
The enhancement of generalization in robots by large vision-language models (LVLMs) is increasingly evident. Therefore, the embodied cognitive abilities of LVLMs based on egocentric videos are of great interest. However, current datasets for embodied video question answering lack comprehensive and systematic evaluation frameworks. Critical embodied cognitive issues, such as robotic self-cognition, dynamic scene perception, and hallucination, are rarely addressed. To tackle these challenges, we propose ECBench, a high-quality benchmark designed to systematically evaluate the embodied cognitive abilities of LVLMs. ECBench features a diverse range of scene video sources, open and varied question formats, and 30 dimensions of embodied cognition. To ensure quality, balance, and high visual dependence, ECBench uses class-independent meticulous human annotation and multi-round question screening strategies. Additionally, we introduce ECEval, a comprehensive evaluation system that ensures the fairness and rationality of the indicators. Utilizing ECBench, we conduct extensive evaluations of proprietary, open-source, and task-specific LVLMs. ECBench is pivotal in advancing the embodied cognitive capabilities of LVLMs, laying a solid foundation for developing reliable core models for embodied agents. All data and code are available at https://github.com/Rh-Dang/ECBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
