arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.07445cs.CL
收藏

SIB-200:一个面向200+种语言和方言主题分类的简单、包容且大规模的评估数据集

SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects

University College London(伦敦大学学院);University of Toronto(多伦多大学);Masakhane(马萨卡内);Saarland University(萨尔大学);University of Ontario Institute of Technology(安大略理工大学);Amazon Alexa AI(亚马逊Alexa AI)

David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, Annie En-Shiun Lee

AI总结:

针对多语言NLP评估仅覆盖少数语言、低资源语言NLU评估数据集匮乏的问题,本文构建了覆盖200+种语言和方言的SIB-200主题分类基准数据集,实验发现多语言模型在低资源等语言上性能仍有明显差距,旨在推动更具包容性的多语言模型评估。

摘要:

尽管过去几年我们在多语言自然语言处理领域取得了进展,但评估通常仅限于少数有可用数据集的语言,大量低资源语言被排除在外。本文中,我们构建了SIB-200——一个覆盖200种语言和方言的大规模开源主题分类基准数据集,以解决自然语言理解(NLU)评估数据集匮乏的问题。对于SIB-200覆盖的许多语言而言,这是首个公开可用的NLU评估数据集。该数据集基于Flores-200机器翻译语料库构建,我们对数据集的英文部分进行了标注,并将句子级别的标注扩展到语料库覆盖的其余203种语言。尽管任务简单,我们在全监督设置、跨语言迁移设置以及大语言模型提示设置下的评估表明,当多语言评估扩展到全球众多语言时,高资源语言与低资源语言的性能之间仍存在巨大差距。我们发现,多语言语言模型预训练期间未见过的语言、代表性不足的语系(如尼罗特语系和大西洋-刚果语系),以及非洲、美洲、大洋洲和东南亚地区的语言,在我们的主题分类数据集上的性能通常最低。我们希望该数据集能推动在更多样化的语言集合上对多语言语言模型开展更具包容性的评估。项目地址:https://github.com/dadelani/sib-200

英文摘要:

Despite the progress we have recorded in the last few years in multilingual natural language processing, evaluation is typically limited to a small set of languages with available datasets which excludes a large number of low-resource languages. In this paper, we created SIB-200 -- a large-scale open-sourced benchmark dataset for topic classification in 200 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). For many of the languages covered in SIB-200, this is the first publicly available evaluation dataset for NLU. The dataset is based on Flores-200 machine translation corpus. We annotated the English portion of the dataset and extended the sentence-level annotation to the remaining 203 languages covered in the corpus. Despite the simplicity of this task, our evaluation in full-supervised setting, cross-lingual transfer setting and prompting of large language model setting show that there is still a large gap between the performance of high-resource and low-resource languages when multilingual evaluation is scaled to numerous world languages. We found that languages unseen during the pre-training of multilingual language models, under-represented language families (like Nilotic and Altantic-Congo), and languages from the regions of Africa, Americas, Oceania and South East Asia, often have the lowest performance on our topic classification dataset. We hope our dataset will encourage a more inclusive evaluation of multilingual language models on a more diverse set of languages. https://github.com/dadelani/sib-200

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑