AfroBench：大语言模型在非洲语言上的表现如何？
AfroBench: How Good are Large Language Models on African Languages?
AI总结：
本文提出覆盖64种非洲语言、15项任务和22个数据集的多任务基准 AfroBench，并通过对比提示式 LLM 与 BERT、T5 微调基线，揭示非洲语言性能差距及资源影响。
摘要：
诸如 MEGA 之类的大规模多语言评估通常只包含少数非洲语言，原因在于高质量评估数据稀缺，以及现有非洲数据集的可发现性有限。这种代表性不足阻碍了 LLM 在多样化语言和任务上的全面评估。为应对这些挑战，我们推出 AfroBench——一个多任务基准，用于评估 LLM 在 64 种非洲语言、15 项任务和 22 个数据集上的表现。AfroBench 包含九个自然语言理解数据集、六个文本生成数据集、六项知识与问答任务，以及一项数学推理任务。我们给出了将提示式 LLM 与基于 BERT 和 T5 风格模型的微调基线进行比较的结果。结果表明，在大多数任务中，英语等高资源语言与非洲语言之间存在巨大性能差距；但性能也会随单语数据资源的可用性而变化。研究结果证实，非洲语言上的表现仍是当前 LLM 的障碍，凸显出需要进一步努力以缩小这一差距。项目页面：https://mcgill-nlp.github.io/AfroBench/
英文摘要：
Large-scale multilingual evaluations, such as MEGA, often include only a handful of African languages due to the scarcity of high-quality evaluation data and the limited discoverability of existing African datasets. This lack of representation hinders comprehensive LLM evaluation across a diverse range of languages and tasks. To address these challenges, we introduce AfroBench -- a multi-task benchmark for evaluating the performance of LLMs across 64 African languages, 15 tasks and 22 datasets. AfroBench consists of nine natural language understanding datasets, six text generation datasets, six knowledge and question answering tasks, and one mathematical reasoning task. We present results comparing the performance of prompting LLMs to fine-tuned baselines based on BERT and T5-style models. Our results suggest large gaps in performance between high-resource languages, such as English, and African languages across most tasks; but performance also varies based on the availability of monolingual data resources. Our findings confirm that performance on African languages continues to remain a hurdle for current LLMs, underscoring the need for additional efforts to close this gap. https://mcgill-nlp.github.io/AfroBench/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
