ChiKhaPo: 一个用于评估大型语言模型词汇理解与生成能力的大规模多语言基准
ChiKhaPo: A Large-Scale Multilingual Benchmark for Evaluating Lexical Comprehension and Generation in Large Language Models
AI总结：
针对现有基准语言覆盖不足且侧重高阶任务的问题，提出ChiKhaPo基准，包含8个子任务，覆盖2700+种语言，评估LLM的词汇理解与生成能力，发现6个SOTA模型表现不佳。
摘要：
现有的大型语言模型（LLM）基准主要局限于高资源或中资源语言，并且通常评估推理和生成方面的高阶任务性能。然而，大量证据表明，LLM在全球3800多种书面语言中的绝大多数语言中缺乏基本的语言能力。我们引入了ChiKhaPo，它包含8个难度不同的子任务，旨在评估生成模型的词汇理解和生成能力。ChiKhaPo利用现有的词典、单语数据和双语文本，为2个子任务提供了2700多种语言的覆盖，在语言覆盖范围上超过了任何现有基准。我们进一步展示了6个SOTA模型在我们的基准上表现不佳，并讨论了影响性能分数的因素，包括语系、语言资源丰富度、任务以及理解与生成方向。通过ChiKhaPo，我们希望促进并鼓励对LLM进行大规模多语言基准测试。
英文摘要：
Existing benchmarks for large language models (LLMs) are largely restricted to high- or mid-resource languages, and often evaluate performance on higher-order tasks in reasoning and generation. However, plenty of evidence points to the fact that LLMs lack basic linguistic competence in the vast majority of the world's 3800+ written languages. We introduce ChiKhaPo, consisting of 8 subtasks of varying difficulty designed to evaluate the lexical comprehension and generation abilities of generative models. ChiKhaPo draws on existing lexicons, monolingual data, and bitext, and provides coverage for 2700+ languages for 2 subtasks, surpassing any existing benchmark in terms of language coverage. We further show that 6 SOTA models struggle on our benchmark, and discuss the factors contributing to performance scores, including language family, language resourcedness, task, and comprehension versus generation directions. With ChiKhaPo, we hope to enable and encourage the massively multilingual benchmarking of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
