Natural Language Processing (NLP) helps empower intelligent machines by enhancing a better understanding of the human language for linguistic-based human-computer communication. Recent developments in computational power and the advent of large amounts of linguistic data have heightened the need and demand for automating semantic analysis using data-driven approaches. The utilization of data-driven strategies is pervasive now due to the significant improvements demonstrated through the usage of deep learning methods in areas such as Computer Vision, Automatic Speech Recognition, and in particular, NLP. This survey categorizes and addresses the different aspects and applications of NLP that have benefited from deep learning. It covers core NLP tasks and applications and describes how deep learning methods and models advance these areas. We further analyze and compare different approaches and state-of-the-art models.
Raytheon BBN Technologies（雷神BBN技术公司）；Harvard University（哈佛大学）；University of Pennsylvania（宾夕法尼亚大学）；University of Oregon（俄勒冈大学）；University of the Basque Country (UPV/EHU)（巴斯克大学 (UPV/EHU)）
Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.
As an effective strategy, data augmentation (DA) alleviates data scarcity scenarios where deep learning techniques may fail. It is widely applied in computer vision then introduced to natural language processing and achieves improvements in many tasks. One of the main focuses of the DA methods is to improve the diversity of training data, thereby helping the model to better generalize to unseen testing data. In this survey, we frame DA methods into three categories based on the diversity of augmented data, including paraphrasing, noising, and sampling. Our paper sets out to analyze DA methods …
National Taiwan University（国立台湾大学）；Amazon AI（亚马逊人工智能）；University of Stuttgart（斯图加特大学）
Deep learning has been the mainstream technique in natural language processing (NLP) area. However, the techniques require many labeled data and are less generalizable across domains. Meta-learning is an arising field in machine learning studying approaches to learn better learning algorithms. Approaches aim at improving algorithms in various aspects, including data efficiency and generalizability. Efficacy of approaches has been shown in many NLP tasks, but there is no systematic survey of these approaches in NLP, which hinders more researchers from joining the field. Our goal with this surve…
A Survey on Model Compression for Large Language Models
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）
Graph Learning and Its Advancements on Large Language Models: A Holistic Survey
School of Business Administration, Southwestern University of Finance and Economics（西南财经大学工商管理学院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Zhongguancun Laboratory（中关村实验室）；Meta；University of Electronic Science and Technology of China（电子科技大学）；Institute of Digital Economy and Interdisciplinary Science Innovation, Southwestern University of Finance and Economics（西南财经大学数字经济与跨学科科学创新研究院）
As NLP models become more integrated with the everyday lives of people, it becomes important to examine the social effect that the usage of these systems has. While these models understand language and have increased accuracy on difficult downstream tasks, there is evidence that these models amplify gender, racial and cultural stereotypes and lead to a vicious cycle in many settings. In this survey, we analyze the origins of biases, the definitions of fairness, and how different subfields of NLP mitigate bias. We finally discuss how future studies can work towards eradicating pernicious biases…
College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）
How can we design Natural Language Processing (NLP) systems that learn from human feedback? There is a growing research body of Human-in-the-loop (HITL) NLP frameworks that continuously integrate human feedback to improve the model itself. HITL NLP research is nascent but multifarious -- solving various NLP problems, collecting diverse feedback from different people, and applying different methods to learn from collected feedback. We present a survey of HITL NLP work from both Machine Learning (ML) and Human-Computer Interaction (HCI) communities that highlights its short yet inspiring history…
Multi-task learning (MTL) has become increasingly popular in natural language processing (NLP) because it improves the performance of related tasks by exploiting their commonalities and differences. Nevertheless, it is still not understood very well how multi-task learning can be implemented based on the relatedness of training tasks. In this survey, we review recent advances of multi-task learning methods in NLP, with the aim of summarizing them into two general multi-task training methods based on their task relatedness: (i) joint training and (ii) multi-step training. We present examples in…
Continual learning (CL) aims to enable information systems to learn from a continuous data stream across time. However, it is difficult for existing deep learning architectures to learn a new task without largely forgetting previously acquired knowledge. Furthermore, CL is particularly challenging for language learning, as natural language is ambiguous: it is discrete, compositional, and its meaning is context-dependent. In this work, we look at the problem of CL through the lens of various NLP tasks. Our survey discusses major challenges in CL and current methods applied in neural network mod…
Continental-NTU Corporate Lab, Nanyang Technological University（大陆集团-南洋理工大学企业实验室）；Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学苏瑞福公共卫生学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning, Central China Normal University（华中师范大学湖北省人工智能与智慧学习重点实验室）；National Language Resources Monitoring and Research Center for Network Media, Central China Normal University（华中师范大学国家语言资源监测与研究网络媒体中心）；School of Computer Science, Central China Normal University（华中师范大学计算机学院）
Federated Learning: A Cutting-Edge Survey of the Latest Advancements and Applications
Universitat Politecnica de Valencia（瓦伦西亚理工大学）；University of Tehran（德黑兰大学）；University of Houston（休斯顿大学）；Nazarbayev University（纳扎尔巴耶夫大学）；Coventry University（考文垂大学）
Federated Learning aims to learn machine learning models from multiple decentralized edge devices (e.g. mobiles) or servers without sacrificing local data privacy. Recent Natural Language Processing techniques rely on deep learning and large pre-trained language models. However, both big deep neural and language models are trained with huge amounts of data which often lies on the server side. Since text data is widely originated from end users, in this work, we look into recent NLP models and techniques which use federated learning as the learning framework. Our survey discusses major challeng…
LLMs Meet Multimodal Generation and Editing: A Survey
The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）；Tencent AI Lab（腾讯AI实验室）
Foundations and Recent Trends in Multimodal Mobile Agents: A Survey
Australian Artificial Intelligence Institute（澳大利亚人工智能研究所）；The Pennsylvania State University（宾夕法尼亚州立大学）；Beijing Jiaotong University（北京交通大学）；University of Liverpool（利物浦大学）
Queensland University of Technology（昆士兰理工大学）；Information Technology University（信息科技大学）；University of Lincoln（林肯大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；National University of Computer and Emerging Sciences（国家计算机与新兴科学大学）；Qatar University（卡塔尔大学）
From Statistical Methods to Pre-Trained Models; A Survey on Automatic Speech Recognition for Resource Scarce Urdu Language
Shenzhen University（深圳大学）；School of Artificial Intelligence, CAS Institute of Automation（中国科学院自动化研究所人工智能学院）；University of Chinese Academy of Sciences（中国科学院大学）
Comment on "Approaching human language with complex networks" by Cong and Liu (Physics of Life Reviews, Volume 11, Issue 4, December 2014, Pages 598-618).
Effectively scaling large Transformer models is a main driver of recent advances in natural language processing. Dynamic neural networks, as an emerging research direction, are capable of scaling up neural networks with sub-linear increases in computation and time by dynamically adjusting their computational path based on the input. Dynamic neural networks could be a promising solution to the growing parameter numbers of pretrained language models, allowing both model pretraining with trillions of parameters and faster inference on mobile devices. In this survey, we summarize progress of three…
Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing …
Microsoft Research India（微软印度研究院）；Carnegie Mellon University（卡内基梅隆大学）
Code-switching, the alternation of languages within a conversation or utterance, is a common communicative phenomenon that occurs in multilingual communities across the world. This survey reviews computational approaches for code-switched Speech and Natural Language Processing. We motivate why processing code-switched text and speech is essential for building intelligent agents and systems that interact with users in multilingual communities. As code-switching data and resources are scarce, we list what is available in various code-switched language pairs with the language processing tasks the…
Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
Fudan University（复旦大学）；The Chinese University of Hong Kong, MMLab（香港中文大学MMLab）；Lancaster University（兰卡斯特大学）；Tongji University（同济大学）；The University of Toronto（多伦多大学）
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）；Harvard School of Engineering and Applied Sciences（哈佛大学工程与应用科学学院）
The field of natural language processing has seen impressive progress in recent years, with neural network models replacing many of the traditional systems. A plethora of new models have been proposed, many of which are thought to be opaque compared to their feature-rich counterparts. This has led researchers to analyze, interpret, and evaluate neural networks in novel and more fine-grained ways. In this survey paper, we review analysis methods in neural language processing, categorize them according to prominent research trends, highlight existing limitations, and point to potential direction…
Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.
In this work, we provide a survey of active learning (AL) for its applications in natural language processing (NLP). In addition to a fine-grained categorization of query strategies, we also investigate several other important aspects of applying AL to NLP problems. These include AL for structured prediction tasks, annotation cost, model learning (especially with deep neural models), and starting and stopping AL. Finally, we conclude with a discussion of related topics and future directions.
Technical University of Munich（慕尼黑工业大学）；Mila Quebec AI Institute（米拉魁北克人工智能研究所）；McGill University（麦吉尔大学）；King’s College London（伦敦国王学院）
In pace with developments in the research field of artificial intelligence, knowledge graphs (KGs) have attracted a surge of interest from both academia and industry. As a representation of semantic relations between entities, KGs have proven to be particularly relevant for natural language processing (NLP), experiencing a rapid spread and wide adoption within recent years. Given the increasing amount of research work in this area, several KG-related approaches have been surveyed in the NLP research community. However, a comprehensive study that categorizes established topics and reviews the m…
University of Tübingen（蒂宾根大学）；Ghent University（根特大学）；University of Stuttgart（斯图加特大学）
This paper presents a comprehensive survey of corpora and lexical resources available for Turkish. We review a broad range of resources, focusing on the ones that are publicly available. In addition to providing information about the available linguistic resources, we present a set of recommendations, and identify gaps in the data available for conducting research and building applications in Turkish Linguistics and Natural Language Processing.
Saarland University（萨尔兰大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Deep neural networks and huge language models are becoming omnipresent in natural language applications. As they are known for requiring large amounts of training data, there is a growing body of work to improve the performance in low-resource settings. Motivated by the recent fundamental changes towards neural models and the popular pre-train and fine-tune paradigm, we survey promising approaches for low-resource natural language processing. After a discussion about the different dimensions of data availability, we give a structured overview of methods that enable learning when training data …
Northwestern University（西北大学）；The Hong Kong Polytechnic University（香港理工大学）
Irony is a ubiquitous figurative language in daily communication. Previously, many researchers have approached irony from linguistic, cognitive science, and computational aspects. Recently, some progress have been witnessed in automatic irony processing due to the rapid development in deep neural models in natural language processing (NLP). In this paper, we will provide a comprehensive overview of computational irony, insights from linguistic theory and cognitive science, as well as its interactions with downstream NLP tasks and newly proposed multi-X irony processing perspectives.
University of Colorado at Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）
Over the last several years, the field of natural language processing has been propelled forward by an explosion in the use of deep learning models. This survey provides a brief introduction to the field and a quick overview of deep learning architectures and methods. It then sifts through the plethora of recent studies and summarizes a large assortment of relevant contributions. Analyzed research areas include several core linguistic processing issues in addition to a number of applications of computational linguistics. A discussion of the current state of the art is then provided along with …
University of Cambridge（剑桥大学）；MIT（麻省理工学院）；Technion, IIT（以色列理工学院）
In recent years linguistic typology, which classifies the world's languages according to their functional and structural properties, has been widely used to support multilingual NLP. While the growing importance of typological information in supporting multilingual tasks has been recognised, no systematic survey of existing typological resources and their use in NLP has been published. This paper provides such a survey as well as discussion which we hope will both inform and inspire future work in the area.
King Fahd University of Petroleum and Minerals（法赫德国王石油与矿物大学）
Deep learning models usually require a huge amount of data. However, these large datasets are not always attainable. This is common in many challenging NLP tasks. Consider Neural Machine Translation, for instance, where curating such large datasets may not be possible specially for low resource languages. Another limitation of deep learning models is the demand for huge computing resources. These obstacles motivate research to question the possibility of knowledge transfer using large trained models. The demand for transfer learning is increasing as many large models are emerging. In this surv…
The utility and power of Natural Language Processing (NLP) seems destined to change our technological society in profound and fundamental ways. However there are, to date, few accessible descriptions of the science of NLP that have been written for a popular audience, or even for an audience of intelligent, but uninitiated scientists. This paper aims to provide just such an overview. In short, the objective of this article is to describe the purpose, procedures and practical applications of NLP in a clear, balanced, and readable way. We will examine the most recent literature describing the me…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
While end-to-end learning with fully differentiable models has enabled tremendous success in natural language process (NLP) and machine learning, there have been significant recent interests in learning with latent discrete structures to incorporate better inductive biases for improved end-task performance and better interpretability. This paradigm, however, is not straightforwardly amenable to the mainstream gradient-based optimization methods. This work surveys three main families of methods to learn such models: surrogate gradients, continuous relaxation, and marginal likelihood maximizatio…
Delivery Hero SE（Delivery Hero公司）；Delft University of Technology（代尔夫特理工大学）；RISE（RISE研究院）；Boston University（波士顿大学）
Stream processing has been an active research field for more than 20 years, but it is now witnessing its prime time due to recent successful efforts by the research community and numerous worldwide open-source communities. This survey provides a comprehensive overview of fundamental aspects of stream processing systems and their evolution in the functional areas of out-of-order data management, state management, fault tolerance, high availability, load management, elasticity, and reconfiguration. We review noteworthy past research findings, outline the similarities and differences between earl…
