M3Exam：用于检验大型语言模型的多语言、多模态、多级别基准
M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models
AI总结：
M3Exam是一个多语言、多模态、多级别的人类考试基准，含12,317道题目，评估LLMs，发现当前模型包括GPT-4在多语言和复杂多模态任务上仍表现不佳。
摘要：
尽管存在各种用于评估自然语言处理模型的基准，我们认为人类考试是评估大型语言模型（LLMs）通用智能的更合适方式，因为它们本质上要求更广泛的能力，如语言理解、领域知识和问题解决技能。为此，我们引入了M3Exam，这是一个新颖的基准，来源于真实且官方的人类考试题目，用于在多语言、多模态和多级别的背景下评估LLMs。M3Exam具有三个独特特征：（1）多语言性，包含来自多个国家的题目，要求强大的多语言熟练度和文化知识；（2）多模态性，考虑到许多考试题目的多模态性质，以测试模型的多模态理解能力；（3）多级别结构，包含来自三个关键教育阶段的考试，以全面评估模型在不同水平上的熟练度。总共，M3Exam包含12,317道题目，涵盖9种不同语言和三个教育级别，其中约23%的题目需要处理图像才能成功解答。我们评估了顶尖LLMs在M3Exam上的表现，发现当前模型，包括GPT-4，在处理多语言文本时仍然存在困难，尤其是在低资源和非拉丁字母语言方面。多模态LLMs在处理复杂多模态题目时也表现不佳。我们相信M3Exam可以成为全面评估LLMs的宝贵资源，通过检验其多语言和多模态能力并追踪其发展。数据和评估代码可在\url{https://github.com/DAMO-NLP-SG/M3Exam}获取。
英文摘要：
Despite the existence of various benchmarks for evaluating natural language processing models, we argue that human exams are a more suitable means of evaluating general intelligence for large language models (LLMs), as they inherently demand a much wider range of abilities such as language understanding, domain knowledge, and problem-solving skills. To this end, we introduce M3Exam, a novel benchmark sourced from real and official human exam questions for evaluating LLMs in a multilingual, multimodal, and multilevel context. M3Exam exhibits three unique characteristics: (1) multilingualism, encompassing questions from multiple countries that require strong multilingual proficiency and cultural knowledge; (2) multimodality, accounting for the multimodal nature of many exam questions to test the model's multimodal understanding capability; and (3) multilevel structure, featuring exams from three critical educational periods to comprehensively assess a model's proficiency at different levels. In total, M3Exam contains 12,317 questions in 9 diverse languages with three educational levels, where about 23\% of the questions require processing images for successful solving. We assess the performance of top-performing LLMs on M3Exam and find that current models, including GPT-4, still struggle with multilingual text, particularly in low-resource and non-Latin script languages. Multimodal LLMs also perform poorly with complex multimodal questions. We believe that M3Exam can be a valuable resource for comprehensively evaluating LLMs by examining their multilingual and multimodal abilities and tracking their development. Data and evaluation code is available at \url{https://github.com/DAMO-NLP-SG/M3Exam}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
