arXiv 2406.033682025-01-23cs.CLcs.AI

IrokoBench：大型语言模型时代非洲语言的新基准

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

Mila, McGill University（米拉实验室、麦吉尔大学）；Lelapa AI（拉拉帕人工智能公司）；Saarland University（萨尔大学）；University of Toronto（多伦多大学）；University College London（伦敦大学学院）；Microsoft Research Africa（微软研究院非洲分部）；Cohere For AI（科here人工智能）；SADiLaR（南非数字语言资源中心）；Ontario Tech University（安大略理工大学）；Lancaster University（兰卡斯特大学）；Princeton University（普林斯顿大学）；Makerere University（马凯雷雷大学）；Leibniz Universität Hannover（莱布尼茨汉诺威大学）；Le CNAM（法国国立工艺学院）；Imperial College London（伦敦帝国学院）；Universidad de Deusto（德乌斯托大学）；DAUST（达累斯萨拉姆科技大学）；Haramaya University（哈拉马亚大学）

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun… 展开作者

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Tadesse Kebede Guge, Tombekai Vangoni Sherman, Pontus Stenetorp

备注：: Accepted to NAACL 2025 (main conference)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出IrokoBench基准，覆盖17种非洲语言的推理、数学和知识问答任务，评估发现开源与专有模型及高资源与低资源语言间存在显著性能差距。

摘要：

尽管大型语言模型（LLM）被广泛采用，但其卓越能力仍局限于少数高资源语言。此外，许多低资源语言（例如非洲语言）由于缺乏高资源语言之外适当或全面的基准，通常仅在基本的文本分类任务上进行评估。在本文中，我们介绍了IrokoBench——一个针对17种类型多样的低资源非洲语言的人工翻译基准数据集，涵盖三项任务：自然语言推理（AfriXNLI）、数学推理（AfriMGSM）和基于知识的多选题问答（AfriMMLU）。我们使用IrokoBench在10个开源和6个专有LLM上评估了零样本、少样本和翻译测试（即测试集被翻译成英语）设置。我们的评估揭示了高资源语言（如英语和法语）与低资源非洲语言之间的显著性能差距。我们观察到开源与专有模型之间存在显著性能差距，其中性能最佳的开源模型Gemma 2 27B仅达到性能最佳的专有模型GPT-4o性能的63%。此外，在评估前将测试集机器翻译成英语有助于缩小以英语为中心的大型模型（如Gemma 2 27B和LLaMa 3.1 70B）的差距。这些发现表明，需要付出更多努力来开发和适配面向非洲语言的LLM。

英文摘要：

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (\eg African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench -- a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and multi-choice knowledge-based question answering~(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings~(where test sets are translated into English) across 10 open and six proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages~(such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63\% of the best-performing proprietary model GPT-4o performance. In addition, machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, such as Gemma 2 27B and LLaMa 3.1 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。