arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.00765cs.CVcs.AIcs.CL
收藏

MM-Vet v2:一个评估大型多模态模型综合能力的挑战性基准

MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities

National University of Singapore(新加坡国立大学);Microsoft(微软公司);Advanced Micro Devices(超威半导体公司)

Weihao Yu, Zhengyuan Yang, Lingfeng Ren, Linjie Li, Jianfeng Wang, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang, Xinchao Wang

AI总结:

该研究发布 MM-Vet v2 基准,新增图像-文本交错序列理解能力并扩充高质量评估集,用于更全面评测大型多模态模型的综合能力。

摘要:

MM-Vet 以旨在评估综合能力的开放式视觉-语言问题为特色,已成为大型多模态模型评估中最受欢迎的基准之一。MM-Vet 评估六项核心视觉-语言(VL)能力:识别、知识、空间感知、语言生成、OCR 和数学。然而,其问题形式局限于单一图像-文本对,缺乏现实场景中普遍存在的图像与文本交错序列。为解决这一局限,我们提出 MM-Vet v2,其中包含一项名为“图像-文本序列理解”的新型 VL 能力,用于评估模型处理 VL 序列的能力。此外,我们在保持评估样本高质量的同时,进一步扩大了评估集规模。使用 MM-Vet v2 对大型多模态模型进行基准测试后,我们发现 Claude 3.5 Sonnet 是最佳模型,得分为 71.8,略优于得分为 71.0 的 GPT-4o。在开放权重模型中,InternVL2-Llama3-76B 以 68.4 分领先。代码、数据和排行榜可在 https://github.com/yuweihao/MM-Vet 获取。

英文摘要:

MM-Vet, with open-ended vision-language questions targeting at evaluating integrated capabilities, has become one of the most popular benchmarks for large multimodal model evaluation. MM-Vet assesses six core vision-language (VL) capabilities: recognition, knowledge, spatial awareness, language generation, OCR, and math. However, its question format is restricted to single image-text pairs, lacking the interleaved image and text sequences prevalent in real-world scenarios. To address this limitation, we introduce MM-Vet v2, which includes a new VL capability called "image-text sequence understanding", evaluating models' ability to process VL sequences. Furthermore, we maintain the high quality of evaluation samples while further expanding the evaluation set size. Using MM-Vet v2 to benchmark large multimodal models, we found that Claude 3.5 Sonnet is the best model with a score of 71.8, slightly outperforming GPT-4o which scored 71.0. Among open-weight models, InternVL2-Llama3-76B leads with a score of 68.4. The code, data, and leaderboard are accessible at https://github.com/yuweihao/MM-Vet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑