UrduMMLU：乌尔都语理解的大规模多任务基准
UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understanding
AI总结：
针对乌尔都语缺乏本地教育来源的MMLU风格基准，提出包含26,431道多选题的UrduMMLU，覆盖26个学科，评估30个LLM发现Gemini-3.5-Flash最佳，多数模型在人文科目上表现差。
摘要：
有意义的 multilingual 评估必须在目标语言和教育背景下测试模型。乌尔都语有超过2.3亿人使用，但缺乏从本地教育来源构建的广泛MMLU风格基准。我们提出UrduMMLU，一个包含26,431道乌尔都语多选题的基准，涵盖26个学科和五个领域，数据来自本地乌尔都语题库和公开考试PDF。与基于翻译的资源不同，UrduMMLU既包括标准学术科目，也包括乌尔都语和地区特定内容。我们通过双重人工标注和严格共识过滤对考试部分进行标注。我们在英语和乌尔都语提示下评估了30个LLM，进行了60次零样本评估，并进一步在两种提示语言的多个少样本设置下评估了四个开源LLM。Gemini-3.5-Flash表现最佳，准确率达到90.20%和90.34%，而其他模型均未超过85%。最强的开源模型落后7.79和8.92个百分点，许多模型在乌尔都语人文科目上比STEM科目损失25到40个百分点。少样本提示仅带来微小提升。UrduMMLU表明，当前LLM中乌尔都语知识仍不均匀，尤其是地区性内容。
英文摘要：
Meaningful multilingual evaluation must test models in the target language and educational context. Urdu, spoken by more than 230 million people, lacks a broad MMLU-style benchmark built from native educational sources. We introduce UrduMMLU, a benchmark of 26,389 Urdu MCQs across 26 subjects and five domains, collected from native Urdu MCQ banks and public examination PDFs. Unlike translation-based benchmarks, UrduMMLU combines academic subjects with content specific to Urdu and regional education. We label the exam-derived portion through dual human annotation with strict consensus filtering. We evaluate 30 LLMs under English and Urdu prompts, yielding 60 zero-shot evaluations, and further evaluate four open-source LLMs under multiple few-shot settings across both prompt languages. Gemini-3.5-Flash performs best, reaching 90.23% and 90.45% accuracy, while no other model exceeds 85%. The strongest open-source model trails by 7.78 and 9.12 points, and many models lose 25 to 40 points on Urdu-centered Humanities subjects compared with STEM. Few-shot prompting yields only modest gains. Results on UrduMMLU show that current LLMs have uneven Urdu knowledge, particularly for content grounded in the regional context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
