SinhalaMMLU：一个用于评估僧伽罗语多任务语言理解的综合基准
SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala
AI总结：
针对现有LLM评估忽视低资源语言与文化内容、多语言基准翻译易出错的问题，研究推出首个面向僧伽罗语的多任务理解基准SinhalaMMLU，含7000余道贴合斯里兰卡课程的题目，评测26个模型后发现顶尖模型表现仍有限，文化相关领域提升空间大。
摘要：
大语言模型（LLMs）展现出令人瞩目的通用知识与推理能力，但其评估主要聚焦于全球性或以英语为中心的主题，往往忽视低资源语言与文化特定内容。尽管近期的多语言基准试图弥补这一差距，但许多依赖自动翻译，可能引入错误并曲解原有的文化语境。为解决这一问题，我们推出SinhalaMMLU——首个专门为低资源语言僧伽罗语设计的多项选择题问答基准。该数据集包含7000多道题目，覆盖中学到大学教育阶段，与斯里兰卡国家课程对齐，涵盖6个领域和30个学科，既包含通用学术主题，也包含文化根植性知识。我们在SinhalaMMLU上评估了26个LLM，发现尽管Claude 3.5 sonnet和GPT-4o分别以67%和62%的平均准确率取得最高成绩，但模型整体表现仍然有限。尤其是模型在人文学科这类文化内涵丰富的领域表现不佳，表明LLM适配低资源与文化特定语境仍有很大提升空间。
英文摘要：
Large Language Models (LLMs) demonstrate impressive general knowledge and reasoning abilities, yet their evaluation has predominantly focused on global or anglocentric subjects, often neglecting low-resource languages and culturally specific content. While recent multilingual benchmarks attempt to bridge this gap, many rely on automatic translation, which can introduce errors and misrepresent the original cultural context. To address this, we introduce SinhalaMMLU, the first multiple-choice question answering benchmark designed specifically for Sinhala, a low-resource language. The dataset includes over 7,000 questions spanning secondary to collegiate education levels, aligned with the Sri Lankan national curriculum, and covers six domains and 30 subjects, encompassing both general academic topics and culturally grounded knowledge. We evaluate 26 LLMs on SinhalaMMLU and observe that, while Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies at 67% and 62% respectively, overall model performance remains limited. In particular, models struggle in culturally rich domains such as the Humanities, revealing substantial room for improvement in adapting LLMs to low-resource and culturally specific contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
