ArabicMMLU：评估阿拉伯语的大规模多任务语言理解
ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
AI总结：
本文提出了首个阿拉伯语多任务语言理解基准 ArabicMMLU，包含 40 个任务和 14,575 道多项选择题，对 35 个模型的评估揭示了当前阿拉伯语模型性能的巨大改进空间。
摘要：
语言模型评估的重点已转向推理和知识密集型任务，这得益于预训练大型模型的进步。尽管最先进的模型部分基于大量阿拉伯语文本进行训练，但由于相关数据集的可用性有限，评估其在阿拉伯语上的性能仍具挑战。为填补这一空白，我们提出了 ArabicMMLU，这是首个面向阿拉伯语的多任务语言理解基准，数据源自北非、黎凡特和海湾地区不同国家多个教育阶段的学校考试。我们的数据包含 40 个任务和 14,575 道用现代标准阿拉伯语（MSA）编写的多项选择题，并通过与该地区的母语者合作精心构建。我们对 35 个模型进行的全面评估表明，仍有巨大的改进空间，尤其是在最优秀的开源模型中。值得注意的是，BLOOMZ、mT0、LLaMA2 和 Falcon 难以达到 50% 的得分，而即使是表现最好的以阿拉伯语为中心的模型也仅获得了 62.3% 的得分。
英文摘要：
The focus of language model evaluation has transitioned towards reasoning and knowledge-intensive tasks, driven by advancements in pretraining large models. While state-of-the-art models are partially trained on large Arabic texts, evaluating their performance in Arabic remains challenging due to the limited availability of relevant datasets. To bridge this gap, we present \datasetname{}, the first multi-task language understanding benchmark for the Arabic language, sourced from school exams across diverse educational levels in different countries spanning North Africa, the Levant, and the Gulf regions. Our data comprises 40 tasks and 14,575 multiple-choice questions in Modern Standard Arabic (MSA) and is carefully constructed by collaborating with native speakers in the region. Our comprehensive evaluations of 35 models reveal substantial room for improvement, particularly among the best open-source models. Notably, BLOOMZ, mT0, LLaMA2, and Falcon struggle to achieve a score of 50%, while even the top-performing Arabic-centric model only achieves a score of 62.3%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
