BenchMAX：大型语言模型的综合多语言评估套件
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models
AI总结：
针对现有多语言基准忽视高级能力评估的问题，本文提出BenchMAX多语言评估基准，通过多语种数据构建与人工标注，公平比较LLM的指令遵循、推理等核心能力，并揭示模型规模扩大无法弥补的跨语言性能差距。
摘要：
以往的多语言基准主要关注简单的理解任务，但对于大型语言模型（LLMs），我们强调在指令遵循、推理、长上下文理解、代码生成等方面的熟练程度。然而，跨语言衡量这些高级能力的研究尚不充分。为解决这一差距，我们引入了BenchMAX，一个多向多语言评估基准，允许跨语言公平比较这些重要能力。为确保高质量，数据从英语机器翻译成其他16种语言后，每个任务中的每个样本均由三位不同的母语标注者独立标注。此外，我们提出了一个源于数据集构建的新颖翻译挑战。在BenchMAX上进行的大量实验揭示了核心能力在不同语言上的有效性差异，突显了仅通过扩大模型规模无法弥合的性能差距。BenchMAX作为一个全面的多语言评估平台，为促进多语言语言模型的发展提供了一个有前景的测试床。数据集和代码可公开获取。
英文摘要：
Previous multilingual benchmarks focus primarily on simple understanding tasks, but for large language models(LLMs), we emphasize proficiency in instruction following, reasoning, long context understanding, code generation, and so on. However, measuring these advanced capabilities across languages is underexplored. To address the disparity, we introduce BenchMAX, a multi-way multilingual evaluation benchmark that allows for fair comparisons of these important abilities across languages. To maintain high quality, three distinct native-speaking annotators independently annotate each sample within all tasks after the data was machine-translated from English into 16 other languages. Additionally, we present a novel translation challenge stemming from dataset construction. Extensive experiments on BenchMAX reveal varying effectiveness of core capabilities across languages, highlighting performance gaps that cannot be bridged by simply scaling up model size. BenchMAX serves as a comprehensive multilingual evaluation platform, providing a promising test bed to promote the development of multilingual language models. The dataset and code are publicly accessible.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
