MAVERIX：多模态音频视觉评估与识别指数
MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeX
AI总结：
MAVERIX是一个用于评估多模态模型音频视觉整合能力的统一基准，通过精心设计的问题展示模型在跨模态理解上的性能，揭示了与人类水平的显著差距。
摘要：
我们介绍了MAVERIX（多模态音频视觉评估与识别指数），一个统一的基准，用于探测多模态大语言模型中的视频理解能力，涵盖视频、音频和文本输入，并包含人类性能基线。尽管近年来在具有视觉和音频理解能力的模型方面取得了显著进展，但该领域缺乏标准化的评估框架来彻底评估其跨模态理解性能。MAVERIX精选了700个视频中的2556个问题，以多种选择题和开放性问题的形式呈现，专门设计用于通过需要紧密整合视频和音频信息的问题来评估多模态模型，覆盖了广泛的情景。MAVERIX唯一提供音频视觉问题，紧密模拟人类在推理和决策过程中可用的多模态感知经验。据我们所知，MAVERIX是首个专门针对评估此类综合音频视觉整合能力的基准。使用最先进的模型，包括Qwen 2.5 Omni和Gemini 2.5 Flash-Lite进行实验，显示性能约为64%的准确率，而人类专家达到接近天花板的92.8%性能，暴露了与人类水平理解之间显著的差距。通过标准化的评估协议、严格标注的流程和公开的工具包，MAVERIX建立了一个具有挑战性的测试平台，以推动音频视觉多模态智能的发展。
英文摘要：
We introduce MAVERIX (Multimodal audiovisual Evaluation and Recognition IndeX), a unified benchmark to probe the video understanding in multimodal LLMs, encompassing video, audio, text inputs with human performance baselines. Although recent advancements in models with vision and audio understanding capabilities have shown substantial progress, the field lacks a standardized evaluation framework to thoroughly assess their cross-modality comprehension performance. MAVERIX curates 2,556 questions from 700 videos, in the form of both multiple-choice and open-ended formats, explicitly designed to evaluate multimodal models through questions that necessitate tight integration of video and audio information, spanning a broad spectrum of agentic scenarios. MAVERIX uniquely provides models with audiovisual questions, closely mimicking the multimodal perceptual experiences available to humans during inference and decision-making processes. To our knowledge, MAVERIX is the first benchmark aimed explicitly at assessing comprehensive audiovisual integration in such granularity. Experiments with state-of-the-art models, including Qwen 2.5 Omni and Gemini 2.5 Flash-Lite, show performance around 64% accuracy, while human experts reach near-ceiling performance of 92.8%, exposing a substantial gap to human-level comprehension. With standardized evaluation protocols, a rigorously annotated pipeline, and a public toolkit, MAVERIX establishes a challenging testbed for advancing audiovisual multimodal intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
