arXiv 2502.128292025-07-21cs.CL

KazMMLU：评估语言模型在哈萨克语、俄语及哈萨克斯坦区域知识上的表现

KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan

Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·泽亚德人工智能大学）；Institute of Foundation Models（基础模型研究所）

Mukhammed Togmanov, Nurdaulet Mukhituly, Diana Turmakhan, Jonibek Mansurov, Maiya Goloburda, Akhmed Sakip, Zhuohan Xie, Yuxia Wang, Bekassyl Syzdykov, Nurkhan L… 展开作者

Mukhammed Togmanov, Nurdaulet Mukhituly, Diana Turmakhan, Jonibek Mansurov, Maiya Goloburda, Akhmed Sakip, Zhuohan Xie, Yuxia Wang, Bekassyl Syzdykov, Nurkhan Laiyk, Alham Fikri Aji, Ekaterina Kochmar, Preslav Nakov, Fajri Koto

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出首个面向哈萨克语的 MMLU 风格基准 KazMMLU，包含 23,000 道哈俄双语题目，并通过对多款主流多语言模型的评估揭示其在哈萨克语和俄语任务上的显著性能差距。

摘要：

尽管哈萨克斯坦拥有两千万人口，其文化和语言在自然语言处理领域仍然代表性不足。虽然大语言模型（LLMs）在全球持续进步，但哈萨克语的进展有限，这体现在专用模型和基准评估的稀缺上。为填补这一空白，我们提出了 KazMMLU，这是第一个专为哈萨克语设计的 MMLU 风格数据集。KazMMLU 包含 23,000 个问题，涵盖不同教育阶段，包括 STEM、人文学科和社会科学，这些问题来源于真实的教育材料，并由母语者和教育工作者进行人工验证。该数据集包括 10,969 个哈萨克语问题和 12,031 个俄语问题，反映了哈萨克斯坦的双语教育体系和丰富的本地背景。我们对多个最先进的多语言模型（Llama-3.1、Qwen-2.5、GPT-4 和 DeepSeek V3）的评估表明仍有很大的改进空间，因为即使表现最好的模型也难以在哈萨克语和俄语上取得有竞争力的表现。这些发现凸显了与高资源语言相比存在的显著性能差距。我们希望我们的数据集能够推动以哈萨克语为中心的大语言模型的进一步研究与开发。数据和代码将在接收后公开。

英文摘要：

Despite having a population of twenty million, Kazakhstan's culture and language remain underrepresented in the field of natural language processing. Although large language models (LLMs) continue to advance worldwide, progress in Kazakh language has been limited, as seen in the scarcity of dedicated models and benchmark evaluations. To address this gap, we introduce KazMMLU, the first MMLU-style dataset specifically designed for Kazakh language. KazMMLU comprises 23,000 questions that cover various educational levels, including STEM, humanities, and social sciences, sourced from authentic educational materials and manually validated by native speakers and educators. The dataset includes 10,969 Kazakh questions and 12,031 Russian questions, reflecting Kazakhstan's bilingual education system and rich local context. Our evaluation of several state-of-the-art multilingual models (Llama-3.1, Qwen-2.5, GPT-4, and DeepSeek V3) demonstrates substantial room for improvement, as even the best-performing models struggle to achieve competitive performance in Kazakh and Russian. These findings underscore significant performance gaps compared to high-resource languages. We hope that our dataset will enable further research and development of Kazakh-centric LLMs. Data and code will be made available upon acceptance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。