希腊MMLU：用于评估希腊语言模型多任务基准的原生来源基准
GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek
AI总结：
希腊MMLU是一个包含45个学科领域的21805道多选题的希腊多任务评估基准，所有题目均源自希腊学术、专业和政府考试，旨在评估语言模型在希腊语言中的表现。
摘要：
大型语言模型（LLMs）通常在包含希腊语的多语言语料库上进行训练，但针对希腊语的可靠评估基准，尤其是基于真实原生内容的基准，仍然有限。现有数据集通常是从英语机器翻译而来，无法捕捉希腊语言和文化特征。我们介绍了希腊MMLU，一个大规模多任务希腊语言理解基准，包含21,805道多选题，涵盖45个学科领域，按照新定义的学科分类组织，并标注了从初级到专业考试的教育难度级别。所有问题均源自希腊的学术、专业和政府考试。我们公开发布了16,857个样本，并保留4,948个样本用于私人排行榜，以实现稳健且抗污染的评估。对超过80个开源和闭源LLM的评估揭示了前沿模型与开源模型之间以及希腊适应模型与通用多语言模型之间的显著性能差距。最后，我们系统分析了影响性能的因素，包括模型规模、适应和提示，并得出了改进希腊语言中LLM能力的见解。
英文摘要：
Large Language Models (LLMs) are commonly trained on multilingual corpora that include Greek, yet reliable evaluation benchmarks for Greek-particularly those based on authentic, native-sourced content-remain limited. Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. We introduce GreekMMLU, a native-sourced benchmark for massive multitask language understanding in Greek, comprising 21,805 multiple-choice questions across 45 subject areas, organized under a newly defined subject taxonomy and annotated with educational difficulty levels spanning primary to professional examinations. All questions are sourced or authored in Greek from academic, professional, and governmental exams. We publicly release 16,857 samples and reserve 4,948 samples for a private leaderboard to enable robust and contamination-resistant evaluation. Evaluations of over 80 open- and closed-source LLMs reveal substantial performance gaps between frontier and open-weight models, as well as between Greek-adapted models and general multilingual ones. Finally, we provide a systematic analysis of factors influencing performance-including model scale, adaptation, and prompting-and derive insights for improving LLM capabilities in Greek.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
