多语言功能评估用于大语言模型
Multi-lingual Functional Evaluation for Large Language Models
AI总结：
本文提出跨语言小学数学符号基准和跨语言指令跟随评估，通过多语言翻译功能基准模板，评估大语言模型在多语言环境中的性能和鲁棒性，发现CL-GSM Symbolic和CL-IFEval在多语言任务中表现更优。
摘要：
大语言模型中的多语言能力通常通过静态数据基准如Belebele、M-MMLU和M-GSM进行评估。然而，这些评估往往无法充分了解模型在多语言环境中的实际性能和鲁棒性。为此，我们创建了多语言功能基准——跨语言小学数学符号基准（CL-GSM Symbolic）和跨语言指令跟随评估（CL-IFEval），通过将现有功能基准模板从英语翻译成五种额外的语言，这些语言覆盖了NLP中可用的资源范围：法语、西班牙语、印地语、阿拉伯语和约鲁巴语。我们的结果表明，某些静态多语言基准比其他基准更能准确捕捉功能性能（例如，在英语、法语和西班牙语中，M-GSM与CL-GSM Symbolic之间的性能下降分别为24%、17%和18%；同样，Belebele与CL-IFEval之间的性能下降为15-24%，而M-MMLU与CL-IFEval之间的性能下降仅为0.5%-3%）。同样，我们发现模型在不同语言中的鲁棒性差异显著，某些语言（如阿拉伯语、英语）在评估迭代中表现最一致。
英文摘要：
Multi-lingual competence in large language models is often evaluated via static data benchmarks such as Belebele, M-MMLU and M-GSM. However, these evaluations often fail to provide an adequate understanding of the practical performance and robustness of models across multi-lingual settings. In response, we create multi-lingual functional benchmarks -- Cross-Lingual Grade School Math Symbolic (CL-GSM Symbolic) and Cross-Lingual Instruction-Following Eval (CL-IFEval)-- by translating existing functional benchmark templates from English to five additional languages that span the range of resources available for NLP: French, Spanish, Hindi, Arabic and Yoruba. Our results reveal that some static multi-lingual benchmarks capture functional performance much more closely than others (i.e. across models, there is a 24%, 17% and 18% decrease in performance between M-GSM and CL-GSM Symbolic in English, French and Spanish respectively; similarly there's a 15 - 24% performance drop across languages between Belebele and CL-IFEval, and only a 0.5% to 3% performance drop between M-MMLU and CL-IFEval). Similarly, we find that model robustness across languages varies significantly, with certain languages (eg. Arabic, English) being the most consistently well performing across evaluation iterations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
