BnMMLU：测量孟加拉语大规模多任务语言理解
BnMMLU: Measuring Massive Multitask Language Understanding in Bengali
AI总结：
BnMMLU通过大规模多任务评估孟加拉语语言理解，揭示模型推理和应用能力的不足，并推动多语言NLP发展。
摘要：
大规模多任务基准测试推动了语言模型的快速发展，但大多数测试侧重于资源丰富的语言，如英语，导致孟加拉语代表性不足。我们提出了BnMMLU，一个全面的基准测试，用于测量孟加拉语的大规模多任务语言理解。BnMMLU涵盖STEM、人文学科、社会科学和一般知识共41个领域，包含134,375个多选题-选项对——迄今为止最全面的孟加拉语评估套件。该数据集通过MathML保留数学内容，并包含BnMMLU-HARD，一个从最常被顶级系统错误回答的问题中构建的紧凑子集，以强调困难案例。我们对11种LLM家族中的24种模型变体进行了评估，涵盖开放式权重通用/多语言模型、以孟加拉语为中心的开放式权重模型以及专有模型，覆盖多种参数规模和指令微调设置。我们在标准化协议下评估模型，涵盖两种提示风格（直接 vs. 思维链）和两种上下文模式（0次提示 vs. 5次提示），在各家族中报告一致的准确性。我们的分析突显了推理和应用技能的持续差距，并表明模型规模的子线性回报。我们发布数据集和评估模板，以支持严格、可重复的孟加拉语语言理解评估，并促进多语言NLP的进步。
英文摘要：
Large-scale multitask benchmarks have driven rapid progress in language modeling, yet most emphasize high-resource languages such as English, leaving Bengali underrepresented. We present BnMMLU, a comprehensive benchmark for measuring massive multitask language understanding in Bengali. BnMMLU spans 41 domains across STEM, humanities, social sciences, and general knowledge, and contains 134,375 multiple-choice question-option pairs--the most extensive Bengali evaluation suite to date. The dataset preserves mathematical content via MathML, and includes BnMMLU-HARD, a compact subset constructed from questions most frequently missed by top systems to stress difficult cases. We benchmark 24 model variants across 11 LLM families, spanning open-weights general/multilingual, Bengali-centric open-weights, and proprietary models, covering multiple parameter scales and instruction-tuned settings. We evaluate models under standardized protocols covering two prompting styles (Direct vs. Chain-of-Thought) and two context regimes (0-shot vs. 5-shot), reporting accuracy consistently across families. Our analysis highlights persistent gaps in reasoning and application skills and indicates sublinear returns to scale across model sizes. We release the dataset and evaluation templates to support rigorous, reproducible assessment of Bengali language understanding and to catalyze progress in multilingual NLP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
