高 Yao 基准：评估大型语言模型多语言和多文化能力的综合框架
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models
AI总结：
本文提出GaoYao基准，通过182300个样本、26种语言和51个地区，评估大型语言模型的多语言和多文化能力，揭示地理性能差异和任务间差距。
摘要：
评估大型语言模型（LLMs）的多语言和多文化能力对其实用性至关重要。然而，当前基准面临三个关键限制：（1）碎片化的评估维度常忽视深层文化细微差别；（2）依赖低质量机器翻译的主观任务语言覆盖不足；（3）分析浅显，缺乏超越简单排名的诊断深度。为此，我们引入GaoYao，一个包含182300个样本、26种语言和51个地区/区域的综合基准。首先，GaoYao提出一个统一框架，将评估任务分为三个文化层（通用多语言、跨文化、单一文化）和九个认知子层。其次，我们通过专家严格本地化主观基准至19种语言，并合成跨文化测试集至34种文化，覆盖范围超过先前111%。第三，我们对20多种旗舰和紧凑型LLM进行了深入诊断分析。我们的发现揭示了显著的地理性能差异和任务间的明显差距，为未来工作提供了可靠的地图。我们发布了该基准（https://github.com/lunyiliu/GaoYao）
英文摘要：
Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark (https://github.com/lunyiliu/GaoYao).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
