SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects
University College London(伦敦大学学院);University of Toronto(多伦多大学);Masakhane(马萨卡内);Saarland University(萨尔大学);University of Ontario Institute of Technology(安大略理工大学);Amazon Alexa AI(亚马逊Alexa AI)
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, Annie En-Shiun Lee
Despite the progress we have recorded in the last few years in multilingual natural language processing, evaluation is typically limited to a small set of languages with available datasets which excludes a large number of low-resource languages. In this paper, we created SIB-200 -- a large-scale open-sourced benchmark dataset for topic classification in 200 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). For many of the languages covered in SIB-200, this is the first publicly available evaluation dataset for NLU. The dataset is based on Flores-200 machine translation corpus. We annotated the English portion of the dataset and extended the sentence-level annotation to the remaining 203 languages covered in the corpus. Despite the simplicity of this task, our evaluation in full-supervised setting, cross-lingual transfer setting and prompting of large language model setting show that there is still a large gap between the performance of high-resource and low-resource languages when multilingual evaluation is scaled to numerous world languages. We found that languages unseen during the pre-training of multilingual language models, under-represented language families (like Nilotic and Altantic-Congo), and languages from the regions of Africa, Americas, Oceania and South East Asia, often have the lowest performance on our topic classification dataset. We hope our dataset will encourage a more inclusive evaluation of multilingual language models on a more diverse set of languages. https://github.com/dadelani/sib-200
Alexandria: A Multi-Domain Dialectal Arabic Machine Translation Dataset for Culturally Inclusive and Linguistically Diverse LLMs
The University of British Columbia(不列颠哥伦比亚大学);Canada Research Chair in NLP and ML(自然语言处理和机器学习研究主席);Mohammed VI Polytechnic University(穆莱·阿卜杜勒阿齐兹国王理工学院);Birzeit University(比尔泽特大学);Western Michigan University(西部密歇根大学);Tuwaiq Academy(图瓦伊克学院);King Khalid University(国王卡利德大学);American University of Beirut(贝鲁特美国大学);Ibb University(伊卜大学);University of Hail(海勒大学);University of Technology and Applied Sciences(技术与应用科学大学);Arab Open University(阿拉伯开放大学);Minia University(米尼亚大学);Nantes University(南特大学);Prince Sultan University(沙特王子大学);Umm Al-Qura University(乌姆·阿勒·卡拉大学);University of Nouakchott(努尔人大学);Fatabyyano(法塔比亚诺);Independent Researcher(独立研究者);Hadhramout University(哈德拉姆大学);Cairo University(开罗大学);Misurata University(米斯拉塔大学);Al-Balqa Applied University(巴勒斯坦应用大学);University of Khartoum(喀土穆大学);Sultan Qaboos Higher Centre for Culture and Science(穆罕默德·本·拉希德·阿勒马克图姆文化与科学高级中心);Arab Center for Research and Policy Studies(阿拉伯研究中心);Hamad Bin Khalifa University(哈马德·本·哈利法大学);Institut Supérieur du Numérique(数字高级学院)
Ibom NLP: A Step Toward Inclusive Natural Language Processing for Nigeria's Minority Languages
Electrical Engineering and Computer Science, Howard University(电气工程与计算机科学系,霍华德大学);AIMS Research and Innovation Centre(AIMS研究与创新中心);NM-AIST;Mila - Quebec AI Institute(魁北克AI研究所);McGill University(麦吉尔大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);Korapay;National Institute for Nigerian Languages(尼日利亚语言国家研究所);University of Uyo(乌耶大学)
RegSpeech12: A Regional Corpus of Bengali Spontaneous Speech Across Dialects
BRAC University(孟加拉国布拉克大学);Bangladesh University of Engineering and Technology(孟加拉国工程技术大学);Shahjalal University of Science and Technology(沙赫贾拉勒科技大学);Khulna University(库尔纳大学);Islamic University of Technology(伊斯兰科技大学);Daffodil International University(达福迪尔国际大学);Boston University(波士顿大学);Rice University(莱斯大学)
Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs
The University of British Columbia(不列颠哥伦比亚大学);MBZUAI(穆扎伊布人工智能研究所);Invertible AI(可逆人工智能);Birzeit University(比尔泽特大学);Prince Sultan University(沙特王储大学);UM6P;Cairo University(开罗大学);JUST(朱拉大学);Ain Shams University(艾因·夏姆斯大学);Damascus University(大马士革大学);University of Khartoum(喀土穆大学);Menoufiya University(蒙努菲亚大学);University of Nouakchott(努瓦克肖特大学);National Polytechnic School of Algiers(阿尔及利亚国家多科技术学校);Full Sail University(全帆大学);Alfaisal University(阿尔法西大学);Hamad Bin Khalifa University(哈利法大学)
PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects
Shenzhen International Graduate School, Tsinghua University(深圳国际研究生院,清华大学);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);JD AI Research(京东人工智能研究院)
AfriVoices-KE: A Multilingual Speech Dataset for Kenyan Languages
Maseno University(马塞诺大学);United States International University(美国国际大学);Dedan Kimathi University of Technology(丹金athi科技大学);Kabarak University(卡巴拉克大学);Jaramogi Oginga Odinga University of Science and Technology(贾拉摩吉奥金加大学科学技术学院);KU Leuven(库勒芬大学);Tech Innovators Network (THiNK)(技术创新者网络(THiNK))