arXiv 2412.009482024-12-01cs.CL

Uhura：评估低资源非洲语言中科学问答与真实性的基准

Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages

Imperial College London（伦敦帝国学院）；University of South Africa（南非大学）；University of Johannesburg（约翰内斯堡大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席项目）；Masakhane General Purpose（Masakhane通用项目）；Saarland University（萨尔大学）；Equiano Institute（埃夸诺研究所）；OpenAI（OpenAI公司）

Edward Bayes, Israel Abebe Azime, Jesujoba O. Alabi, Jonas Kgomo, Tyna Eloundou, Elizabeth Proehl, Kai Chen, Imaan Khadir, Naome A. Etori, Shamsuddeen Hassan Mu… 展开作者

Edward Bayes, Israel Abebe Azime, Jesujoba O. Alabi, Jonas Kgomo, Tyna Eloundou, Elizabeth Proehl, Kai Chen, Imaan Khadir, Naome A. Etori, Shamsuddeen Hassan Muhammad, Choice Mpanza, Igneciah Pocia Thete, Dietrich Klakow, David Ifeoluwa Adelani

备注：: working paper

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出覆盖六种非洲语言的 Uhura 科学问答与真实性基准，揭示专有模型优于开源模型且英语表现普遍更好，并开源基准与平台。

摘要：

大型语言模型（LLM）在知识密集型任务和事实准确性方面的评估往往主要聚焦于高资源语言，原因在于低资源语言（LRL）的数据集十分稀缺。本文提出 Uhura——一个新的基准，通过人工翻译现有英文基准构建，涵盖六种类型多样的非洲语言上的两项任务。第一个数据集 Uhura-ARC-Easy 由多项选择科学问题组成。第二个数据集 Uhura-TruthfulQA 是一个安全基准，测试模型在健康、法律、金融和政治等主题上的真实性。我们重点说明了为低资源语言创建具有高度技术性内容的基准所面临的挑战，并概述了缓解策略。评估显示，GPT-4o、o1-preview 和 Claude 模型等专有模型，与 Meta 的 LLaMA 和 Google 的 Gemma 等开源模型之间存在显著性能差距。此外，所有模型在英语上的表现均优于非洲语言。结果表明，语言模型在回答科学问题时存在困难，并且在低资源非洲语言中更容易生成错误陈述。我们的发现强调，有必要持续提升低资源语言场景下的多语言语言模型能力，以确保其在现实环境中安全、可靠地使用。我们开源了 Uhura Benchmark 和 Uhura Platform，以促进低资源语言自然语言处理（NLP）的进一步研究与开发。

英文摘要：

Evaluations of Large Language Models (LLMs) on knowledge-intensive tasks and factual accuracy often focus on high-resource languages primarily because datasets for low-resource languages (LRLs) are scarce. In this paper, we present Uhura -- a new benchmark that focuses on two tasks in six typologically-diverse African languages, created via human translation of existing English benchmarks. The first dataset, Uhura-ARC-Easy, is composed of multiple-choice science questions. The second, Uhura-TruthfulQA, is a safety benchmark testing the truthfulness of models on topics including health, law, finance, and politics. We highlight the challenges creating benchmarks with highly technical content for LRLs and outline mitigation strategies. Our evaluation reveals a significant performance gap between proprietary models such as GPT-4o and o1-preview, and Claude models, and open-source models like Meta's LLaMA and Google's Gemma. Additionally, all models perform better in English than in African languages. These results indicate that LMs struggle with answering scientific questions and are more prone to generating false claims in low-resource African languages. Our findings underscore the necessity for continuous improvement of multilingual LM capabilities in LRL settings to ensure safe and reliable use in real-world contexts. We open-source the Uhura Benchmark and Uhura Platform to foster further research and development in NLP for LRLs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。