IndicMMLU-Pro：在多任务语言理解上对印度语系大语言模型进行基准测试
IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding
AI总结：
本文提出了IndicMMLU-Pro基准，基于MMLU Pro框架评估大语言模型在9种印度次大陆主要语言的多任务理解能力，涵盖语言理解、推理与生成任务，并提供了最先进多语言模型的基线结果。
摘要：
印度语系在印度次大陆拥有超过15亿使用者，由于其丰富的文化遗产、语言多样性和复杂的结构，为自然语言处理（NLP）研究带来了独特的挑战与机遇。IndicMMLU-Pro是一个旨在跨印度语系评估大语言模型（LLMs）的综合基准，它建立在MMLU Pro（大规模多任务语言理解）框架之上。我们的基准涵盖印地语、孟加拉语、古吉拉特语、马拉地语、卡纳达语、旁遮普语、泰米尔语、泰卢固语和乌尔都语等主要语言，应对了印度次大陆语言多样性所带来的独特挑战与机遇。该基准包含语言理解、推理和生成方面的广泛任务，经过精心设计以捕捉印度语言的复杂性。IndicMMLU-Pro提供了一个标准化评估框架，旨在推动印度语系AI的研究边界，促进开发更准确、高效且具备文化敏感性的模型。本文概述了该基准的设计原则、任务分类体系和数据收集方法，并展示了来自最先进多语言模型的基线结果。
英文摘要：
Known by more than 1.5 billion people in the Indian subcontinent, Indic languages present unique challenges and opportunities for natural language processing (NLP) research due to their rich cultural heritage, linguistic diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark designed to evaluate Large Language Models (LLMs) across Indic languages, building upon the MMLU Pro (Massive Multitask Language Understanding) framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique challenges and opportunities presented by the linguistic diversity of the Indian subcontinent. This benchmark encompasses a wide range of tasks in language comprehension, reasoning, and generation, meticulously crafted to capture the intricacies of Indian languages. IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. This paper outlines the benchmarks' design principles, task taxonomy, and data collection methodology, and presents baseline results from state-of-the-art multilingual models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
