WorldSense: 评估多模态大语言模型的现实世界多模态理解
WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs
AI总结：
WorldSense是首个评估多模态大语言模型现实世界多模态理解能力的基准，通过多模态协作、多样化视频任务和高质量标注，全面评估模型在复杂场景中的表现。
摘要：
我们介绍了WorldSense，第一个评估多模态视频理解的基准，同时涵盖了视觉、音频和文本输入。与现有基准相比，我们的WorldSense具有几个特点：（i）多模态的协作，我们设计的评估任务强调音频和视频的强耦合，要求模型有效利用多模态的协同感知；（ii）视频和任务的多样性，WorldSense包含1662个同步音频视频，系统分类为8个主要领域和67个细粒度子类别以覆盖广泛场景，并包含3172个多选QA对，覆盖26个不同任务，以实现全面评估；（iii）高质量的标注，所有QA对均由80名专家标注者进行多轮修正以确保质量。基于我们的WorldSense，我们对各种最先进的模型进行了广泛评估。实验结果表明，现有模型在理解现实世界场景方面面临显著挑战（最佳准确率为65.1%）。通过分析当前模型的局限性，我们旨在提供有价值的见解，以指导现实世界理解的发展。我们希望WorldSense能为评估从多模态构建和理解连贯上下文的能力提供一个平台。
英文摘要：
We introduce WorldSense, the first benchmark to assess the multi-modal video understanding, that simultaneously encompasses visual, audio, and text inputs. In contrast to existing benchmarks, our WorldSense has several features: (i)collaboration of omni-modality, we design the evaluation tasks to feature a strong coupling of audio and video, requiring models to effectively utilize the synergistic perception of omni-modality; (ii)diversity of videos and tasks, WorldSense encompasses a diverse collection of 1,662 audio-visual synchronised videos, systematically categorized into 8 primary domains and 67 fine-grained subcategories to cover the broad scenarios, and 3,172 multi-choice QA pairs across 26 distinct tasks to enable the comprehensive evaluation; (iii)high-quality annotations, all the QA pairs are manually labeled by 80 expert annotators with multiple rounds of correction to ensure quality. Based on our WorldSense, we extensively evaluate various state-of-the-art models. The experimental results indicate that existing models face significant challenges in understanding real-world scenarios (65.1% best accuracy). By analyzing the limitations of current models, we aim to provide valuable insight to guide development of real-world understanding. We hope our WorldSense can provide a platform for evaluating the ability in constructing and understanding coherent contexts from omni-modality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
