arXiv 2407.124022024-10-03cs.CL

TurkishMMLU：衡量土耳其语大规模多任务语言理解能力

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

Technical University of Munich（慕尼黑工业大学）；Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）

Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, Hinrich Schütze

备注：: EMNLP 2024 - Findings

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出首个土耳其语多任务多项选择基准 TurkishMMLU，包含超1万道由课程专家编写的高中课程题目，覆盖9个科目，并通过对20多个 LLMs 的零样本、少样本及思维链评估，系统揭示当前模型在土耳其语理解上的能力与局限。

摘要：

多项选择问答任务评估大型语言模型（LLMs）的推理、理解和数学能力。虽然现有基准测试采用自动翻译进行多语言评估，但这种方法容易出错，并且可能引入带有文化偏见的问题，尤其是在社会科学领域。我们引入了第一个多任务、多项选择的土耳其语问答基准测试 TurkishMMLU，用于评估 LLMs 对土耳其语的理解能力。TurkishMMLU 包含超过 10,000 个问题，涵盖土耳其高中教育课程中的 9 个不同科目。这些问题由课程专家编写，适合土耳其高中课程，涵盖从自然科学和数学问题到更具文化代表性的主题，如土耳其文学和土耳其共和国历史。我们评估了超过 20 个 LLMs，包括多语言开源模型（如 Gemma、Llama、MT5）、闭源模型（GPT 4o、Claude、Gemini）以及土耳其语适配模型（如 Trendyol）。我们提供了广泛的评估，包括 LLMs 的零样本和少样本评估、思维链推理、问题难度分析以及模型性能。我们对当前 LLMs 的土耳其语能力和局限性进行了深入分析，为未来面向土耳其语的 LLMs 提供见解。我们公开发布了数据集和评估代码：https://github.com/ArdaYueksel/TurkishMMLU。

英文摘要：

Multiple choice question answering tasks evaluate the reasoning, comprehension, and mathematical abilities of Large Language Models (LLMs). While existing benchmarks employ automatic translation for multilingual evaluation, this approach is error-prone and potentially introduces culturally biased questions, especially in social sciences. We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs' understanding of the Turkish language. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models. We provide an extensive evaluation, including zero-shot and few-shot evaluation of LLMs, chain-of-thought reasoning, and question difficulty analysis along with model performance. We provide an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language. We publicly release our code for the dataset and evaluation: https://github.com/ArdaYueksel/TurkishMMLU.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。