arXiv 2404.167902024-04-25cs.CV

SEED-Bench-2-Plus：基于富文本视觉理解的多模态大语言模型基准测试

SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension

Tencent AI Lab（腾讯AI实验室）；School of Data Science, The Chinese University of HongKong, Shenzhen（香港中文大学（深圳）数据科学学院）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）

Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, Ying Shan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SEED-Bench-2-Plus基准测试，包含2.3K道多项选择题，用于评估多模态大语言模型在图表、地图和网页等富文本场景下的视觉理解能力，并揭示了34个主流模型的局限性。

摘要：

理解富文本视觉内容对于多模态大语言模型（MLLMs）的实际应用至关重要，因为富文本场景在现实世界中无处不在，其特征是图像中嵌入了大量文本。近期，具备令人印象深刻通用性的MLLMs的出现，提高了我们对MLLMs的期望。然而，它们在富文本场景中的能力尚未得到全面且客观的评估，因为当前的MLLM基准测试主要侧重于评估通用视觉理解。在这项工作中，我们引入了SEED-Bench-2-Plus，这是一个专门为评估MLLMs的富文本视觉理解而设计的基准测试。我们的基准测试包含2.3K道带有精确人工标注的多项选择题，涵盖三大类别：Charts、Maps和Webs，每个类别都覆盖了现实世界中广泛的富文本场景。这些类别由于其固有的复杂性和多样性，有效地模拟了现实世界的富文本环境。我们进一步对34个主流MLLMs（包括GPT-4V、Gemini-Pro-Vision和Claude-3-Opus）进行了全面评估，并强调了当前MLLMs在富文本视觉理解方面的局限性。我们希望我们的工作能作为现有MLLM基准测试的有益补充，提供有见地的观察，并激发对MLLMs富文本视觉理解领域的进一步研究。数据集和评估代码可在https://github.com/AILab-CVC/SEED-Bench获取。

英文摘要：

Comprehending text-rich visual content is paramount for the practical application of Multimodal Large Language Models (MLLMs), since text-rich scenarios are ubiquitous in the real world, which are characterized by the presence of extensive texts embedded within images. Recently, the advent of MLLMs with impressive versatility has raised the bar for what we can expect from MLLMs. However, their proficiency in text-rich scenarios has yet to be comprehensively and objectively assessed, since current MLLM benchmarks primarily focus on evaluating general visual comprehension. In this work, we introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating \textbf{text-rich visual comprehension} of MLLMs. Our benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs, each of which covers a wide spectrum of text-rich scenarios in the real world. These categories, due to their inherent complexity and diversity, effectively simulate real-world text-rich environments. We further conduct a thorough evaluation involving 34 prominent MLLMs (including GPT-4V, Gemini-Pro-Vision and Claude-3-Opus) and emphasize the current limitations of MLLMs in text-rich visual comprehension. We hope that our work can serve as a valuable addition to existing MLLM benchmarks, providing insightful observations and inspiring further research in the area of text-rich visual comprehension with MLLMs. The dataset and evaluation code can be accessed at https://github.com/AILab-CVC/SEED-Bench.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。