Aligning with ACL 2022 special Theme on "Language Diversity: from Low Resource to Endangered Languages", we discuss the major linguistic and sociopolitical challenges facing development of NLP technologies for African languages. Situating African languages in a typological framework, we discuss how the particulars of these languages can be harnessed. To facilitate future research, we also highlight current efforts, communities, venues, datasets, and tools. Our main objective is to motivate and advocate for an Afrocentric approach to technology development. With this in mind, we recommend \textit{what} technologies to build and \textit{how} to build, evaluate, and deploy them based on the needs of local African communities.
The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP
University of California, Los Angeles(加州大学洛杉矶分校);Georgia Institute of Technology(佐治亚理工学院);University of Wisconsin - Madison(威斯康星大学麦迪逊分校);University of Cape Coast(科克伍德大学);Carleton University(卡尔顿大学);Stetson University(史蒂文森大学);Northwestern University in Qatar(卡塔尔西北大学);Cornell University(康奈尔大学);Soka University of America(美国立命大学);Columbia University(哥伦比亚大学)
AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languages
Princeton University(普林斯顿大学);Laboratory for Artificial Intelligence, Princeton University(普林斯顿大学人工智能实验室);Gaston Berger University(加斯顿·伯杰大学);Mila, McGill University(麦吉尔大学米拉研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能教席);Paris Nanterre University(巴黎南泰尔大学);Paris-Saclay University(巴黎-萨克雷大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);LORIA(洛林计算机科学实验室);Université de Lorraine(洛林大学);University of Trento(特伦托大学);University of Minnesota–Twin Cities(明尼苏达大学双城分校);Imperial College London(伦敦帝国学院);Binghamton University(宾汉姆顿大学);Makerere University(马凯雷雷大学);Penn State University(宾夕法尼亚州立大学);Mbarara University of Science and Technology(姆巴拉拉科技大学);Chalmers University of Technology(查尔姆斯理工大学);University of Ibadan(伊巴丹大学);Nnamdi Azikiwe University(纳姆迪·阿齐基韦大学);South African Centre for Digital Language Resources(南非数字语言资源中心)
AfroBench: How Good are Large Language Models on African Languages?
Masakhane NLP(Masakhane自然语言处理);Mila - Quebec AI Institute & McGill University(Mila-魁北克人工智能研究所及麦吉尔大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);Lelapa AI(Lelapa人工智能);The African Research Collective(非洲研究集体);University of Waterloo(滑铁卢大学);University College London(伦敦大学学院)
The State of Large Language Models for African Languages: Progress and Challenges
University of Gondar(贡德大学);AIMS Research and Innovation Centre(AIMS研究与创新中心);Bahir Dar University(巴赫达尔大学);Bayero University Kano(卡诺巴耶罗大学);Addis Ababa University(亚的斯亚贝巴大学);Imperial College London(伦敦帝国学院);University of Hamburg(汉堡大学)
Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding
Salesforce AI Research(赛富时人工智能研究院);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead
Saarland University, Saarland Informatics Campus(萨尔兰大学,萨尔兰信息学校区);LMU Munich and Munich Center for Machine Learning(慕尼黑大学及慕尼黑机器学习中心);Mila - Quebec AI Institute, McGill University & Canada CIFAR AI Chair(魁北克人工智能研究所,麦吉尔大学及加拿大 CIFAR 人工智能 chair)
The Rise of AfricaNLP: A Survey of Contributions, Contributors, Community Impact, and Bibliometric Analysis
Instituto Politécnico Nacional(国立理工学院);University of Gondar(贡德尔大学);Northwest University Kano(卡诺西北大学);Bayero University Kano(巴耶罗大学卡诺分校);University of Huddersfield(哈德斯菲尔德大学);iCog Labs(iCog实验室);Pennsylvania State University(宾夕法尼亚州立大学);University of Pretoria(比勒陀利亚大学);Masakhane;University of Hamburg(汉堡大学);Imperial College London(伦敦帝国理工学院)
Adapting to the Low-Resource Double-Bind: Investigating Low-Compute Methods on Low-Resource African Languages
Lelapa AI(莱拉帕人工智能公司);Luleå University of Technology(吕勒奥理工大学);Lanfrica(兰弗里卡);Instituto Politécnico Nacional(墨西哥国立理工学院);USC Information Sciences Institute(南加州大学信息科学研究所);University of California, Davis(加州大学戴维斯分校);New Mexico State University(新墨西哥州立大学);Technical University of Munich(慕尼黑工业大学);University of Porto(波尔图大学);ML Collective(机器学习集体);University of Toronto(多伦多大学);University of Dayton(代顿大学);McGill University(麦吉尔大学);Mila Quebec AI Institute(米拉魁北克人工智能研究所)
Mafoko: Structuring and Building Open Multilingual Terminologies for South African NLP
Data Science for Social Impact, Dept. of Computer Science, University of Pretoria(南非普里尼亚大学计算机科学系数据科学与社会影响研究中心);AfriDSAI, University of Pretoria(南非普里尼亚大学非洲数据科学与人工智能研究所);Lelapa AI(莱拉帕人工智能);Economics and Management Sciences, University of the Free State(南非自由州大学经济与管理科学系);Interdisciplinary Centre for Digital Futures, University of the Free State(南非自由州大学跨学科数字未来研究中心)
MasakhaNER: Named Entity Recognition for African Languages
Saarland University(萨尔大学);Retro Rabbit(Retro Rabbit公司);Carnegie Mellon University(卡内基梅隆大学);ProQuest(ProQuest公司);Google Research(谷歌研究院);Brandeis University(布兰迪斯大学);DeepMind(DeepMind公司);Duolingo(Duolingo公司);University of Tsukuba(筑波大学);University of Porto(波尔图大学);Bayero University, Kano(巴耶罗大学);Technical University of Munich(慕尼黑工业大学);Makerere University(马凯雷雷大学);African Leadership University(非洲领导力大学);University of Lagos(拉各斯大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Universität Hamburg(汉堡大学);University of Chinese Academy of Science(中国科学院大学);Lancaster University(兰卡斯特大学);University of Electronic Science and Technology of China(电子科技大学);Luleå University of Technology(吕勒奥理工大学);African University of Science and Technology(非洲科技大学);University of Ibadan(伊巴丹大学);Namibia University of Science and Technology(纳米比亚科技大学);Instadeep(Instadeep公司);Jacobs University Bremen(不来梅雅各布大学);University of Waterloo(滑铁卢大学)
HausaMovieReview: A Benchmark Dataset for Sentiment Analysis in Low-Resource African Language
Federal University Dutsin-Ma(杜钦马联邦大学);Aliko Dangote University of Science and Technology(阿利科·丹格特科技大学);Northwest University(西北大学);Bayero University Kano(卡诺巴耶罗大学)
AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages
Mila Quebec AI Institute(米拉魁北克人工智能研究所);McGill University(麦吉尔大学);Instituto Politécnico Nacional(国家理工学院);Uppsala University(乌普萨拉大学);Universidad de Deusto(德乌斯托大学);Ashesi University(阿希西大学);Montclair State University(蒙特克莱尔州立大学);The College of Saint Rose(圣罗斯学院);Technical University of Munich(慕尼黑工业大学)
An assessment of orthographic similarity measures for several African languages
University of Cape Town(开普敦大学)
众所周知,自然语言界面和工具,如拼写检查和以自己的语言进行的网络搜索,在以信通技术为媒介的通信中是有用的。然而,南部非洲的大多数语言资源不足。因此,如果通用的和少数特定于语言的NLP工具都可以重用或轻松地跨语言调整,这将是非常有用的。这取决于语言之间相似性的概念和程度。我们从正字法和语料库的角度来评估这一点。12个版本的《世界人权宣言》(UDHR)进行了检查,显示集群的语言,从而或多或少地适合跨语言适应的NLP工具,这不符合格思里区。为了检验这些结果的普遍性,我们放大isiZulu定量和定性与其他四个语料库和不同类型的文本。结果表明,《世界人权宣言》是一个典型的文本文件的正字法。研究结果还提供了对词汇多样性和体裁等典型指标可用性的深入了解,以及相同的统计数据在不同的文档中可能意味着不同的东西。虽然NLTK for Python可以用于文本的基本分析,但它和类似的NLP工具需要大量的定制。
Correcting FLORES Evaluation Dataset for Four African Languages
University of Pretoria(比勒陀利亚大学);Council for Scientific and Industrial Research(科学与工业研究理事会);Imperial College(帝国理工学院);Northeastern University(东北大学);University of KwaZulu-Natal(夸祖鲁-纳塔尔大学);Lelapa AI(莱拉帕人工智能公司)
Masakhane;Africa(非洲);The State of African NLP(非洲自然语言处理现状(注:此处“The State of African NLP”更可能是论文章节标题而非机构,根据提取规则应忽略,原HTML中作者机构仅明确出现“Masakhane”“masakhane.io”“Africa”,但“Africa”为地理概念非机构,故仅保留“Masakhane”和“masakhane.io”为机构)}]}(注:经再次核查,原HTML中作者机构明确为“Affiliation: Masakhane, Africa”,“Affiliation: masakhane.io”,其中“Africa”属于地理概念非机构,因此仅保留“Masakhane”和“masakhane.io”为作者所属机构,修正后输出应为{)
AI4D Africa(AI4D非洲);Takwimu Lab(Takwimu实验室);Knowledge for All Foundation(知识共享基金会);Retro Rabbit(Retro Rabbit公司);University of Pretoria(比勒陀利亚大学);Jomo Kenyatta University of Agriculture and Technology(乔莫·肯雅塔农业技术大学);UNESCO(联合国教科文组织);Saarland University(萨尔大学);The University of Malawi(马拉维大学);Baamtu Datamation(Baamtu Datamation公司);TAVODET Youth Development(TAVODET青年发展组织);iCompass(iCompass公司)
AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
McGill University(麦吉尔大学);Mila-Quebec AI Institute(魁北克AI研究所);LMU Munich & Munich Center for Machine Learning(慕尼黑大学及慕尼黑机器学习中心);Microsoft AI for Good Research Lab(微软AI for Good研究实验室)
AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages
University of Toronto(多伦多大学);Mila - Quebec AI Institute, McGill University(魁北克AI研究院,麦吉尔大学);Saarland University, Saarland Informatic Campus(萨尔兰大学,萨尔兰信息学院);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages
Masakhane;University of Pretoria(比勒陀利亚大学);Niger-Volta LTI;African Masters in Machine Intelligence(非洲机器智能硕士项目);Federal University of Technology, Akure(阿库雷联邦理工大学);University of Johannesburg(约翰内斯堡大学);Bayero University, Kano(卡诺贝罗大学);Jomo Kenyatta University of Agriculture and Technology(乔莫·肯雅塔农业技术大学);UESTC(电子科技大学);Siseng Consulting Ltd(Siseng咨询有限公司);African Leadership University(非洲领导力大学);InstaDeep Ltd(InstaDeep有限公司);Sapienza University of Rome(罗马第一大学);Udacity(优达城);Parliament, Republic of South Africa(南非共和国议会);Explore Data Science Academy(Explore数据科学学院);UCD, Konta;AI4D
MasakhaNEWS: News Topic Classification for African languages
University College London(伦敦大学学院);Saarland University(萨尔大学);Instituto Politécnico Nacional(国立理工学院);Lelapa AI(Lelapa人工智能);Fudan University(复旦大学);University of Waterloo(滑铁卢大学);McGill University(麦吉尔大学);Mila Quebec AI Institute(米拉魁北克人工智能研究所);Lanfrica;Technical University of Munich(慕尼黑工业大学);Luleå University of Technology(吕勒奥理工大学);Tanzania Data Lab(坦桑尼亚数据实验室);Makerere University(马凯雷雷大学);Insight Centre for Data Analytics(洞察数据分析中心);Paderborn University(帕德博恩大学);Aston University(阿斯顿大学);University of Porto(波尔图大学);Dire Dawa University(德雷达瓦大学);Kaduna State University(卡杜纳州立大学);Ahmadu Bello University(艾哈迈德·贝罗大学)
本文提出覆盖 16 种非洲语言的新闻主题分类基准数据集 MasakhaNEWS,并评估经典机器学习、语言模型微调及零样本和少样本方法,发现 ChatGPT 提示和 PET 在低资源场景下表现突出。