Recent advances in word embeddings and language models use large-scale, unlabelled data and self-supervised learning to boost NLP performance. Multilingual models, often trained on web-sourced data like Wikipedia, face challenges: few low-resource languages are included, their data is often noisy, and lack of labeled datasets makes it hard to evaluate performance outside high-resource languages like English. In this dissertation, we focus on languages spoken in Sub-Saharan Africa where all the indigenous languages in this region can be regarded as low-resourced in terms of the availability of labelled data for NLP tasks and unlabelled data found on the web. We analyse the noise in the publicly available corpora, and curate a high-quality corpus, demonstrating that the quality of semantic representations learned in word embeddings does not only depend on the amount of data but on the quality of pre-training data. We demonstrate empirically the limitations of word embeddings, and the opportunities the multilingual pre-trained language model (PLM) offers especially for languages unseen during pre-training and low-resource scenarios. We further study how to adapt and specialize multilingual PLMs to unseen African languages using a small amount of monolingual texts. To address the under-representation of the African languages in NLP research, we developed large scale human-annotated labelled datasets for 21 African languages in two impactful NLP tasks: named entity recognition and machine translation. We conduct an extensive empirical evaluation using state-of-the-art methods across supervised, weakly-supervised, and transfer learning settings.
Lelapa AI（莱拉帕人工智能公司）；Luleå University of Technology（吕勒奥理工大学）；Lanfrica（兰弗里卡）；Instituto Politécnico Nacional（墨西哥国立理工学院）；USC Information Sciences Institute（南加州大学信息科学研究所）；University of California, Davis（加州大学戴维斯分校）；New Mexico State University（新墨西哥州立大学）；Technical University of Munich（慕尼黑工业大学）；University of Porto（波尔图大学）；ML Collective（机器学习集体）；University of Toronto（多伦多大学）；University of Dayton（代顿大学）；McGill University（麦吉尔大学）；Mila Quebec AI Institute（米拉魁北克人工智能研究所）
Many natural language processing (NLP) tasks make use of massively pre-trained language models, which are computationally expensive. However, access to high computational resources added to the issue of data scarcity of African languages constitutes a real barrier to research experiments on these languages. In this work, we explore the applicability of low-compute approaches such as language adapters in the context of this low-resource double-bind. We intend to answer the following question: do language adapters allow those who are doubly bound by data and compute to practically build useful m…
The Rise of AfricaNLP: A Survey of Contributions, Contributors, Community Impact, and Bibliometric Analysis
Instituto Politécnico Nacional（国立理工学院）；University of Gondar（贡德尔大学）；Northwest University Kano（卡诺西北大学）；Bayero University Kano（巴耶罗大学卡诺分校）；University of Huddersfield（哈德斯菲尔德大学）；iCog Labs（iCog实验室）；Pennsylvania State University（宾夕法尼亚州立大学）；University of Pretoria（比勒陀利亚大学）；Masakhane；University of Hamburg（汉堡大学）；Imperial College London（伦敦帝国理工学院）
The State of Large Language Models for African Languages: Progress and Challenges
University of Gondar（贡德大学）；AIMS Research and Innovation Centre（AIMS研究与创新中心）；Bahir Dar University（巴赫达尔大学）；Bayero University Kano（卡诺巴耶罗大学）；Addis Ababa University（亚的斯亚贝巴大学）；Imperial College London（伦敦帝国学院）；University of Hamburg（汉堡大学）
Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead
Saarland University, Saarland Informatics Campus（萨尔兰大学，萨尔兰信息学校区）；LMU Munich and Munich Center for Machine Learning（慕尼黑大学及慕尼黑机器学习中心）；Mila - Quebec AI Institute, McGill University & Canada CIFAR AI Chair（魁北克人工智能研究所，麦吉尔大学及加拿大 CIFAR 人工智能 chair）
AfroBench: How Good are Large Language Models on African Languages?
Masakhane NLP（Masakhane自然语言处理）；Mila - Quebec AI Institute & McGill University（Mila-魁北克人工智能研究所及麦吉尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）；Lelapa AI（Lelapa人工智能）；The African Research Collective（非洲研究集体）；University of Waterloo（滑铁卢大学）；University College London（伦敦大学学院）
IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
Mila, McGill University（米拉实验室、麦吉尔大学）；Lelapa AI（拉拉帕人工智能公司）；Saarland University（萨尔大学）；University of Toronto（多伦多大学）；University College London（伦敦大学学院）；Microsoft Research Africa（微软研究院非洲分部）；Cohere For AI（科here人工智能）；SADiLaR（南非数字语言资源中心）；Ontario Tech University（安大略理工大学）；Lancaster University（兰卡斯特大学）；Princeton University（普林斯顿大学）；Makerere University（马凯雷雷大学）；Leibniz Universität Hannover（莱布尼茨汉诺威大学）；Le CNAM（法国国立工艺学院）；Imperial College London（伦敦帝国学院）；Universidad de Deusto（德乌斯托大学）；DAUST（达累斯萨拉姆科技大学）；Haramaya University（哈拉马亚大学）
Rejected Dialects: Biases Against African American Language in Reward Models
Carnegie Mellon University（卡内基梅隆大学）；Columbia University（哥伦比亚大学）；Instituto Superior Técnico, University of Lisbon（里斯本大学高等技术学院）；Instituto de Telecomunicações（电信研究所）
AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages
Mila Quebec AI Institute（米拉魁北克人工智能研究所）；McGill University（麦吉尔大学）；Instituto Politécnico Nacional（国家理工学院）；Uppsala University（乌普萨拉大学）；Universidad de Deusto（德乌斯托大学）；Ashesi University（阿希西大学）；Montclair State University（蒙特克莱尔州立大学）；The College of Saint Rose（圣罗斯学院）；Technical University of Munich（慕尼黑工业大学）
Data Caricatures: On the Representation of African American Language in Pretraining Corpora
Columbia University, Department of Computer Science（哥伦比亚大学计算机科学系）；University of Michigan, Department of Linguistics（密歇根大学语言学系）；University of Pennsylvania, School of Social Policy and Practice, Annenberg School for Communications（宾夕法尼亚大学社会政策与实践学院、安纳伯格通讯学院）；University of Tennessee, Knoxville, Department of English（田纳西大学基洛纳分校英语系）
LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering
York University（约克大学）；Vector Institute（向量研究所）；Connected Minds（连接思维公司）；Emory University（埃默里大学）；Wilfrid Laurier University（威尔弗里德·劳里埃大学）；University of Toronto（多伦多大学）；University of Guelph（圭尔夫大学）；Monark Health（莫纳克健康公司）；CIFAR Solution Network（加拿大高级研究院解决方案网络）
We highlight an important frontier in algorithmic fairness: disparity in the quality of natural language processing algorithms when applied to language from authors of different social groups. For example, current systems sometimes analyze the language of females and minorities more poorly than they do of whites and males. We conduct an empirical analysis of racial disparity in language identification for tweets written in African-American English, and discuss implications of disparity in NLP.
Jacobs University Bremen（不来梅雅各布大学）；University of Khartoum（喀土穆大学）
From Word2Vec to GloVe, word embedding models have played key roles in the current state-of-the-art results achieved in Natural Language Processing. Designed to give significant and unique vectorized representations of words and entities, those models have proven to efficiently extract similarities and establish relationships reflecting semantic and contextual meaning among words and entities. African Languages, representing more than 31% of the worldwide spoken languages, have recently been subject to lots of research. However, to the best of our knowledge, there are currently very few to non…
Faced with a considerable lack of resources in African languages to carry out work in Natural Language Processing (NLP), Natural Language Understanding (NLU) and artificial intelligence, the research teams of NTeALan association has set itself the objective of building open-source platforms for the collaborative construction of lexicographic data in African languages. In this article, we present our first reports after 2 years of collaborative construction of lexicographic resources useful for African NLP tools.
Natural language processing is a branch of computer science that combines artificial intelligence with linguistics. It aims to analyze a language element such as writing or speaking with software and convert it into information. Considering that each language has its own grammatical rules and vocabulary diversity, the complexity of the studies in this field is somewhat understandable. For instance, Turkish is a very interesting language in many ways. Examples of this are agglutinative word structure, consonant/vowel harmony, a large number of productive derivational morphemes (practically infi…
Modality is one of the important components of grammar in linguistics. It lets speaker to express attitude towards, or give assessment or potentiality of state of affairs. It implies different senses and thus has different perceptions as per the context. This paper presents an account showing the gap in the functionality of the current state of art Natural Language Processing (NLP) systems. The contextual nature of linguistic modality is studied. In this paper, the works and logical approaches employed by Natural Language Processing systems dealing with modality are reviewed. It sees human cog…
Reproducing results in publications by distributing publicly available source code is becoming ever more popular. Given the difficulty of reproducing machine learning (ML) experiments, there have been significant efforts in reducing the variance of these results. As in any science, the ability to consistently reproduce results effectively strengthens the underlying hypothesis of the work, and thus, should be regarded as important as the novel aspect of the research itself. The contribution of this work is a framework that is able to reproduce consistent results and provides a means of easily c…
University of Potsdam（波茨坦大学）；Department of Linguistics（语言学系）
"This paper introduces a new task and a new dataset", "we improve the state of the art in X by Y" -- it is rare to find a current natural language processing paper (or AI paper more generally) that does not contain such statements. What is mostly left implicit, however, is the assumption that this necessarily constitutes progress, and what it constitutes progress towards. Here, we make more precise the normally impressionistically used notions of language task and language game and ask how a research programme built on these might make progress towards the goal of modelling general language co…
Institute for Datability Science（数据可及性科学研究所）；Osaka University（大阪大学）；Graduate School of Dentistry（牙医学研究生院）
We work on the task of automatically designing a treatment plan from the findings included in the medical certificate written by the dentist. To develop an artificial intelligence system that deals with free-form certificates written by dentists, we annotate the findings and utilized the natural language processing approach. As a result of the experiment using 990 certificates, 0.585 F1-score was achieved for the task of extracting orthodontic problems from findings, and 0.584 correlation coefficient with the human ranking was achieved for the treatment prioritization task.
Dual decomposition, and more generally Lagrangian relaxation, is a classical method for combinatorial optimization; it has recently been applied to several inference problems in natural language processing (NLP). This tutorial gives an overview of the technique. We describe example algorithms, describe formal guarantees for the method, and describe practical issues in implementing the algorithms. While our examples are predominantly drawn from the NLP literature, the material should be of general relevance to inference problems in machine learning. A central theme of this tutorial is that Lagr…
Many search systems work with large amounts of natural language data, e.g., search queries, user profiles and documents, where deep learning based natural language processing techniques (deep NLP) can be of great help. In this paper, we introduce a comprehensive study of applying deep NLP techniques to five representative tasks in search engines. Through the model design and experiments of the five tasks, readers can find answers to three important questions: (1) When is deep NLP helpful/not helpful in search systems? (2) How to address latency challenges? (3) How to ensure model robustness? T…
We introduce an NLP toolkit based on object-oriented knowledge base and multi-level grammar base. This toolkit focuses on semantic parsing, it also has abilities to discover new knowledge and grammar automatically, new discovered knowledge and grammar will be identified by human, and will be used to update the knowledge base and grammar base. This process can be iterated many times to improve the toolkit continuously.
Natural Language Processing (NLP) has evolved significantly over the last decade. This paper highlights the most important milestones of this period while trying to pinpoint the contribution of each individual model and algorithm to the overall progress. Furthermore, it focuses on issues still remaining to be solved, emphasizing the groundbreaking proposals of Transformers, BERT, and all the similar attention-based models.
We utilise Richards-Engelhardt framework as a tool for understanding Natural Language Processing systems diagrams. Through four examples from scholarly proceedings, we find that the application of the framework to this ecological and complex domain is effective for reflecting on these diagrams. We argue for vocabulary to describe multiple-codings, semiotic variability, and inconsistency or misuse of visual encoding principles in diagrams. Further, for application to scholarly Natural Language Processing systems, and perhaps systems diagrams more broadly, we propose the addition of "Grouping by…
In natural language processing (NLP), labeling on regions of text, such as words, sentences and paragraphs, is a basic task. In this paper, label is defined as map between mention of entity in a region on text and context of entity in a broader region on text containing the mention. This definition naturally introduces linkage of entities induced from inclusion relation of regions, and connected entities form a graph representing information flow defined by map. It also enables calculation of information loss through map using entropy, and entropy lost is regarded as distance between two entit…
In this article, we first give an introduction to artificial intelligence and its applications in biology and medicine in Section 1. Deep learning methods are then described in Section 2. We narrow down the focus of the study on textual data in Section 3, where natural language processing and its applications in the biomedical domain are described. In Section 4, we give an introduction to explainable artificial intelligence and discuss the importance of explainability of artificial intelligence systems, especially in the biomedical domain.
Our aim is to extract information about literary characters in unstructured texts. We employ natural language processing and reasoning on domain ontologies. The first task is to identify the main characters and the parts of the story where these characters are described or act. We illustrate the system in a scenario in the folktale domain. The system relies on a folktale ontology that we have developed based on Propp's model for folktales morphology.
Federated Learning aims to learn machine learning models from multiple decentralized edge devices (e.g. mobiles) or servers without sacrificing local data privacy. Recent Natural Language Processing techniques rely on deep learning and large pre-trained language models. However, both big deep neural and language models are trained with huge amounts of data which often lies on the server side. Since text data is widely originated from end users, in this work, we look into recent NLP models and techniques which use federated learning as the learning framework. Our survey discusses major challeng…
Currently, natural language processing (NLP) models proliferate language discrimination leading to potentially harmful societal impacts as a result of biased outcomes. For example, part-of-speech taggers trained on Mainstream American English (MAE) produce non-interpretable results when applied to African American English (AAE) as a result of language features not seen during training. In this work, we incorporate a human-in-the-loop paradigm to gain a better understanding of AAE speakers' behavior and their language use, and highlight the need for dialectal language inclusivity so that native…
CNRS（法国国家科学研究中心）；École normale supérieure（巴黎高等师范学院）；Université Sorbonne Nouvelle（索邦新大学）；University of Potsdam（波茨坦大学）
This special issue is dedicated to get a better picture of the relationships between computational linguistics and cognitive science. It specifically raises two questions: "what is the potential contribution of computational language modeling to cognitive science?" and conversely: "what is the influence of cognitive science in contemporary computational linguistics?"
Raytheon BBN Technologies（雷神BBN技术公司）；Harvard University（哈佛大学）；University of Pennsylvania（宾夕法尼亚大学）；University of Oregon（俄勒冈大学）；University of the Basque Country (UPV/EHU)（巴斯克大学 (UPV/EHU)）
Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.
Effectively scaling large Transformer models is a main driver of recent advances in natural language processing. Dynamic neural networks, as an emerging research direction, are capable of scaling up neural networks with sub-linear increases in computation and time by dynamically adjusting their computational path based on the input. Dynamic neural networks could be a promising solution to the growing parameter numbers of pretrained language models, allowing both model pretraining with trillions of parameters and faster inference on mobile devices. In this survey, we summarize progress of three…
Rochester Institute of Technology（罗切斯特理工学院）；Centre National Collaboratif de l’Education en Robotique et en Intelligence Artificielle (RobotsMali)（国家机器人与人工智能教育合作中心（马里机器人））；Google Research（谷歌研究院）；Orange Silicon Valley（奥兰治硅谷）
Low-resource languages present unique challenges to (neural) machine translation. We discuss the case of Bambara, a Mande language for which training data is scarce and requires significant amounts of pre-processing. More than the linguistic situation of Bambara itself, the socio-cultural context within which Bambara speakers live poses challenges for automated processing of this language. In this paper, we present the first parallel data set for machine translation of Bambara into and from English and French and the first benchmark results on machine translation to and from Bambara. We discus…
Neuroscience Institute（神经科学研究所）；Department of Machine Learning（机器学习系）；Carnegie Mellon University（卡内基梅隆大学）
Neural networks models for NLP are typically implemented without the explicit encoding of language rules and yet they are able to break one performance record after another. This has generated a lot of research interest in interpreting the representations learned by these networks. We propose here a novel interpretation approach that relies on the only processing system we have that does understand language: the human brain. We use brain imaging recordings of subjects reading complex natural text to interpret word and sequence embeddings from 4 recent NLP models - ELMo, USE, BERT and Transform…
This paper relates work done during the DiLAF project. It consists in converting 5 bilingual African language-French dictionaries originally in Word format into XML following the LMF model. The languages processed are Bambara, Hausa, Kanuri, Tamajaq and Songhai-zarma, still considered as under-resourced languages concerning Natural Language Processing tools. Once converted, the dictionaries are available online on the Jibiki platform for lookup and modification. The DiLAF project is first presented. A description of each dictionary follows. Then, the conversion methodology from .doc format to …
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Reliable uncertainty quantification is a first step towards building explainable, transparent, and accountable artificial intelligent systems. Recent progress in Bayesian deep learning has made such quantification realizable. In this paper, we propose novel methods to study the benefits of characterizing model and data uncertainties for natural language processing (NLP) tasks. With empirical experiments on sentiment analysis, named entity recognition, and language modeling using convolutional and recurrent neural network models, we show that explicitly modeling uncertainties is not only necess…
Singapore University of Technology and Design（新加坡科技设计大学）
We review the scholarly contributions that utilise Natural Language Processing (NLP) techniques to support the design process. Using a heuristic approach, we gathered 223 articles that are published in 32 journals within the period 1991-present. We present state-of-the-art NLP in-and-for design research by reviewing these articles according to the type of natural language text sources: internal reports, design concepts, discourse transcripts, technical publications, consumer opinions, and others. Upon summarizing and identifying the gaps in these contributions, we utilise an existing design in…
This document concerns data readiness in the context of machine learning and Natural Language Processing. It describes how an organization may proceed to identify, make available, validate, and prepare data to facilitate automated analysis methods. The contents of the document is based on the practical challenges and frequently asked questions we have encountered in our work as an applied research institute with helping organizations and companies, both in the public and private sectors, to use data in their business processes.
Technical University of Munich（慕尼黑工业大学）；Jacobs University（雅各布斯大学）
Over the years, there have been campaigns to include the African languages in the growing research on machine translation (MT) in particular, and natural language processing (NLP) in general. Africa has the highest language diversity, with 1500-2000 documented languages and many more undocumented or extinct languages(Lewis, 2009; Bendor-Samuel, 2017). This makes it hard to keep track of the MT research, models and dataset that have been developed for some of them. As the internet and social media make up the daily lives of more than half of the world(Lin, 2020), as well as over 40% of Africans…
We propose a formalism for representation of finite languages, referred to as the class of IDL-expressions, which combines concepts that were only considered in isolation in existing formalisms. The suggested applications are in natural language processing, more specifically in surface natural language generation and in machine translation, where a sentence is obtained by first generating a large set of candidate sentences, represented in a compact way, and then by filtering such a set through a parser. We study several formal properties of IDL-expressions and compare this new formalism with m…
Virtual assistants and text chatbots have recently been gaining popularity. Given the short message nature of text-based chat interactions, the language identification systems of these bots might only have 15 or 20 characters to make a prediction. However, accurate text language identification is important, especially in the early stages of many multilingual natural language processing pipelines. This paper investigates the use of a naive Bayes classifier, to accurately predict the language family that a piece of text belongs to, combined with a lexicon based classifier to distinguish the spec…
This paper describes the system of storage, extract and processing of information structured similarly to the natural language. For recursive inference the system uses the rules having the same representation, as the data. The environment of storage of information is provided with the File Mapping (SHM) mechanism of operating system. In the paper the main principles of construction of dynamic data structure and language for record of the inference rules are stated; the features of available implementation are considered and the description of the application realizing semantic information retr…
