EXAMS-V：一个用于评估视觉语言模型的多学科多语言多模态考试基准
EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models
AI总结：
EXAMS-V是一个包含20,932道多模态考题、覆盖20个学科和11种语言的新基准，旨在评估视觉语言模型，其高难度对GPT-4V等先进模型构成挑战。
摘要：
我们介绍了EXAMS-V，这是一个新的具有挑战性的多学科、多语言、多模态考试基准，用于评估视觉语言模型。它包含20,932道多项选择题，涵盖20个学校学科，涉及自然科学、社会科学以及其他杂项研究，例如宗教、美术、商业等。EXAMS-V包含了多种多模态特征，如文本、图像、表格、图形、图表、地图、科学符号和方程式。这些题目来自7个语系的11种语言。与现有基准不同，EXAMS-V通过收集来自不同国家、具有不同教育体系的学校考试题目而独特地策划而成。这种独特的方法要求在不同语言之间进行复杂的推理，并依赖于特定区域的知识。解决数据集中的问题需要对文本和图像的视觉内容进行高级感知和联合推理。我们的评估结果表明，这是一个具有挑战性的数据集，即使对于GPT-4V和Gemini等先进的视觉文本模型来说也很困难；这强调了数据集的内在复杂性及其作为未来基准的重要性。
英文摘要：
We introduce EXAMS-V, a new challenging multi-discipline multimodal multilingual exam benchmark for evaluating vision language models. It consists of 20,932 multiple-choice questions across 20 school disciplines covering natural science, social science, and other miscellaneous studies, e.g., religion, fine arts, business, etc. EXAMS-V includes a variety of multimodal features such as text, images, tables, figures, diagrams, maps, scientific symbols, and equations. The questions come in 11 languages from 7 language families. Unlike existing benchmarks, EXAMS-V is uniquely curated by gathering school exam questions from various countries, with a variety of education systems. This distinctive approach calls for intricate reasoning across diverse languages and relies on region-specific knowledge. Solving the problems in the dataset requires advanced perception and joint reasoning over the text and the visual content of the image. Our evaluation results demonstrate that this is a challenging dataset, which is difficult even for advanced vision-text models such as GPT-4V and Gemini; this underscores the inherent complexity of the dataset and its significance as a future benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
