A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications
Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）；University of Chinese Academy of Sciences（中国科学院大学）；George Mason University（乔治梅森大学）；Chinese Academy of Sciences（中国科学院）
Pretrained Language Models (PLMs) such as BERT have revolutionized the landscape of Natural Language Processing (NLP). Inspired by their proliferation, tremendous efforts have been devoted to Pretrained Graph Models (PGMs). Owing to the powerful model architectures of PGMs, abundant knowledge from massive labeled and unlabeled graph data can be captured. The knowledge implicitly encoded in model parameters can benefit various downstream tasks and help to alleviate several fundamental issues of learning on graphs. In this paper, we provide the first comprehensive survey for PGMs. We firstly present the limitations of graph representation learning and thus introduce the motivation for graph pre-training. Then, we systematically categorize existing PGMs based on a taxonomy from four different perspectives. Next, we present the applications of PGMs in social recommendation and drug discovery. Finally, we outline several promising research directions that can serve as a guideline for future research.
University of Strathclyde（斯特拉斯克莱德大学）；University of Abertay Dundee（阿伯泰邓迪大学）
Intrusion detection has attracted a considerable interest from researchers and industries. The community, after many years of research, still faces the problem of building reliable and efficient IDS that are capable of handling large quantities of data, with changing patterns in real time situations. The work presented in this manuscript classifies intrusion detection systems (IDS). Moreover, a taxonomy and survey of shallow and deep networks intrusion detection systems is presented based on previous and current works. This taxonomy and survey reviews machine learning techniques and their perf…
In the meantime, a wide variety of terminologies, motivations, approaches, and evaluation criteria have been developed within the research field of explainable artificial intelligence (XAI). With the amount of XAI methods vastly growing, a taxonomy of methods is needed by researchers as well as practitioners: To grasp the breadth of the topic, compare methods, and to select the right XAI method based on traits required by a specific use-case context. Many taxonomies for XAI methods of varying level of detail and depth can be found in the literature. While they often have a different focus, the…
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary fie…
University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）；Tennessee Tech University（田纳西理工大学）；North Carolina A&T State University（北卡罗来纳农工州立大学）
An increasing body of work has recognized the importance of exploiting machine learning (ML) advancements to address the need for efficient automation in extracting access control attributes, policy mining, policy verification, access decisions, etc. In this work, we survey and summarize various ML approaches to solve different access control problems. We propose a novel taxonomy of the ML model's application in the access control domain. We highlight current limitations and open challenges such as lack of public real-world datasets, administration of ML-based access control systems, understan…
National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）；Sumitomo Electric Industries, Ltd.（住友电气工业株式会社）
In this article, we propose the Artificial Intelligence Security Taxonomy to systematize the knowledge of threats, vulnerabilities, and security controls of machine-learning-based (ML-based) systems. We first classify the damage caused by attacks against ML-based systems, define ML-specific security, and discuss its characteristics. Next, we enumerate all relevant assets and stakeholders and provide a general taxonomy for ML-specific threats. Then, we collect a wide range of security controls against ML-specific threats through an extensive review of recent literature. Finally, we classify the…
Goldsmiths, University of London（伦敦大学金史密斯学院）；University of The Arts London（伦敦艺术大学）；Queen Mary University of London（伦敦玛丽女王大学）；Monash University（莫纳什大学）
Generative deep learning systems offer powerful tools for artefact generation, given their ability to model distributions of data and generate high-fidelity results. In the context of computational creativity, however, a major shortcoming is that they are unable to explicitly diverge from the training data in creative ways and are limited to fitting the target data distribution. To address these limitations, there have been a growing number of approaches for optimising, hacking and rewriting these models in order to actively diverge from the training data. We present a taxonomy and comprehensi…
The past few years have seen rapid progress in combining reinforcement learning (RL) with deep learning. Various breakthroughs ranging from games to robotics have spurred the interest in designing sophisticated RL algorithms and systems. However, the prevailing workflow in RL is to learn tabula rasa, which may incur computational inefficiency. This precludes continuous deployment of RL algorithms and potentially excludes researchers without large-scale computing resources. In many other areas of machine learning, the pretraining paradigm has shown to be effective in acquiring transferable know…
Attention is a state of arousal capable of dealing with limited processing bottlenecks in human beings by focusing selectively on one piece of information while ignoring other perceptible information. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. Currently, this property has been widely explored in deep neural networks. Many different neural attention models are now available and have been a very active research area over the past six years. From the theoretical standpoint of attention, this survey provides a critical…
NVIDIA（英伟达）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
The open-world deployment of Machine Learning (ML) algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities such as interpretability, verifiability, and performance limitations. Research explores different approaches to improve ML dependability by proposing new models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks. However, there is a missing connection between ongoing ML research and well-established safety principles. In this paper, we present…
Newcastle University（纽卡斯尔大学）；University of Leeds（利兹大学）；The University of Sydney（悉尼大学）；Cardiff University（卡迪夫大学）；China University of Geoscience (Wuhan)（中国地质大学（武汉））
Machine Learning (ML) and Internet of Things (IoT) are complementary advances: ML techniques unlock complete potentials of IoT with intelligence, and IoT applications increasingly feed data collected by sensors into ML models, thereby employing results to improve their business processes and services. Hence, orchestrating ML pipelines that encompasses model training and implication involved in holistic development lifecycle of an IoT application often leads to complex system integration. This paper provides a comprehensive and systematic survey on the development lifecycle of ML-based IoT appl…
Fraunhofer Center for Machine Learning（弗劳恩霍夫机器学习中心）；Fraunhofer IAIS, Institute for Intelligent Analysis and Information Systems（弗劳恩霍夫智能分析与信息系统研究所）；Fraunhofer SCAI, Institute for Algorithms and Scientific Computing（弗劳恩霍夫算法与科学计算研究所）；Fraunhofer ITWM, Institute for Industrial Mathematics（弗劳恩霍夫工业数学研究所）；Fraunhofer IOSB, Institute for Optronics, System Technologies and Image Exploitation（弗劳恩霍夫光电子学、系统技术与图像开发研究所）
Despite its great success, machine learning can have its limits when dealing with insufficient training data. A potential solution is the additional integration of prior knowledge into the training process which leads to the notion of informed machine learning. In this paper, we present a structured overview of various approaches in this field. We provide a definition and propose a concept for informed machine learning which illustrates its building blocks and distinguishes it from conventional machine learning. We introduce a taxonomy that serves as a classification framework for informed mac…
Southeast University（东南大学）；Purple Mountain Laboratories（紫金山实验室）；Ocean University of China（中国海洋大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Anhui University（安徽大学）；The University of Sydney（悉尼大学）；JD Explore Academy（京东探索研究院）
With the development of convolutional neural networks, hundreds of deep learning based dehazing methods have been proposed. In this paper, we provide a comprehensive survey on supervised, semi-supervised, and unsupervised single image dehazing. We first discuss the physical model, datasets, network modules, loss functions, and evaluation metrics that are commonly used. Then, the main contributions of various dehazing algorithms are categorized and summarized. Further, quantitative and qualitative experiments of various baseline methods are carried out. Finally, the unsolved issues and challeng…
Explainable reinforcement learning (XRL) is an emerging subfield of explainable machine learning that has attracted considerable attention in recent years. The goal of XRL is to elucidate the decision-making process of learning agents in sequential decision-making settings. In this survey, we propose a novel taxonomy for organizing the XRL literature that prioritizes the RL setting. We overview techniques according to this taxonomy. We point out gaps in the literature, which we use to motivate and outline a roadmap for future work.
Università della Svizzera Italiana (USI)（瑞士意大利语区大学）
The growing application of deep neural networks in safety-critical domains makes the analysis of faults that occur in such systems of enormous importance. In this paper we introduce a large taxonomy of faults in deep learning (DL) systems. We have manually analysed 1059 artefacts gathered from GitHub commits and issues of projects that use the most popular DL frameworks (TensorFlow, Keras and PyTorch) and from related Stack Overflow posts. Structured interviews with 20 researchers and practitioners describing the problems they have encountered in their experience have enriched our taxonomy wit…
Large Language Models for EEG: A Comprehensive Survey and Taxonomy
Department of Computer Science and Engineering, Indian Institute of Technology Patna（计算机科学与工程系，印度理工学院帕纳布分校）；Infocomm Technology Cluster, Singapore Institute of Technology（信息通信技术集群，新加坡理工学院）
Andalusian Research Institute in Data Science and Computational Intelligence（安达卢西亚数据科学与计算智能研究院）；University of Granada（格拉纳达大学）
Federated learning is a machine learning paradigm that emerges as a solution to the privacy-preservation demands in artificial intelligence. As machine learning, federated learning is threatened by adversarial attacks against the integrity of the learning model and the privacy of data via a distributed approach to tackle local and global learning. This weak point is exacerbated by the inaccessibility of data in federated learning, which makes harder the protection against adversarial attacks and evidences the need to furtherance the research on defence methods to make federated learning a real…
Clustering is a fundamental machine learning task which has been widely studied in the literature. Classic clustering methods follow the assumption that data are represented as features in a vectorized form through various representation learning techniques. As the data become increasingly complicated and complex, the shallow (traditional) clustering methods can no longer handle the high-dimensional data type. With the huge success of deep learning, especially the deep unsupervised learning, many representation learning techniques with deep architectures have been proposed in the past decade. …
From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation
United Arab Emirates University（阿联酋大学）；Emirates Center for Mobility Research（阿联酋移动性研究中心）
Context: Data Mining (DM) method has been evolving year by year and as of today there is also the enhancement of DM technique that can be run several times faster than the traditional one, called Distributed Data Mining (DDM). It is not a new field in data processing actually, but in the recent years many researchers have been paying more attention on this area. Problems: The number of publication regarding DDM in high reputation journals and conferences has increased significantly. It makes difficult for researchers to gain a comprehensive view of DDM that require further research. Solution: …
Infant Research Centre（婴儿研究中心）；School of Computing（计算学院）；Dublin City University（都柏林城市大学）；ByteDance AI Lab（字节跳动人工智能实验室）；Insight SFI Research Centre for Data Analytics（Insight SFI数据分析研究中心）
Generative adversarial networks (GANs) studies have grown exponentially in the past few years. Their impact has been seen mainly in the computer vision field with realistic image and video manipulation, especially generation, making significant advancements. While these computer vision advances have garnered much attention, GAN applications have diversified across disciplines such as time series and sequence generation. As a relatively new niche for GANs, fieldwork is ongoing to develop high quality, diverse and private time series data. In this paper, we review GAN variants designed for time …
Florida Atlantic University（佛罗里达大西洋大学）；Nanjing University of Finance and Economics（南京财经大学）
Text-to-image synthesis refers to computational methods which translate human written textual descriptions, in the form of keywords or sentences, into images with similar semantic meaning to the text. In earlier research, image synthesis relied mainly on word to image correlation analysis combined with supervised methods to find best alignment of the visual content matching to the text. Recent progress in deep learning (DL) has brought a new set of unsupervised deep learning methods, particularly deep generative models which are able to generate realistic visual images using suitably trained n…
Data Mixing for Large Language Models Pretraining: A Survey and Outlook
The International Joint Institute of Tianjin University（天津大学国际联合研究院）；Tianjin University（天津大学）；TJUNLP Lab, School of Computer Science and Technology（天津大学自然语言处理实验室，计算机科学与技术学院）
Artificial Intelligence (AI) has found its applications in a variety of environments ranging from data science to cybersecurity. AI helps break through the limitations of traditional algorithms and provides more efficient and flexible methods for solving problems. In this paper, we focus on the applications of artificial intelligence in authentication, which is used in a wide range of scenarios including facial recognition to access buildings, keystroke dynamics to unlock smartphones. With the emerging AI-assisted authentication schemes, our comprehensive survey provides an overall understandi…
Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods
The Chinese University of Hong Kong（香港中文大学）；The Hong Kong Polytechnic University（香港理工大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS)（深圳市人工智能与机器人研究院）；Guangdong-Hongkong-Macao Greater Bay Area Weather Research Center for Monitoring Warning and Forecasting (Shenzhen Institute of Meteorological Innovation)（粤港澳大湾区天气监测预警预报研究中心（深圳市气象创新研究院））；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（电子科技大学深圳高等研究院）
A Survey of LLM Prompt Datasets: Taxonomy, Linguistic Patterns, and Practical Uses
School of Computer Science and Technology（计算机科学与技术学院）；Beijing Jiaotong University（北京交通大学）；Department of Computer Science（计算机科学系）；Aalborg University（奥尔堡大学）
Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy
University of Washington（华盛顿大学）；New York University（纽约大学）；Columbia University（哥伦比亚大学）；Liverpool John Moores University（利物浦约翰摩尔大学）；Chennai Mathematical Institute（金奈数学研究所）；Birla Institute of Technology（比拉理工学院）
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM Research AI（IBM人工智能研究院）
Deep learning models, though having achieved great success in many different fields over the past years, are usually data hungry, fail to perform well on unseen samples, and lack of interpretability. Various prior knowledge often exists in the target domain and their use can alleviate the deficiencies with deep learning. To better mimic the behavior of human brains, different advanced methods have been proposed to identify domain knowledge and integrate it into deep models for data-efficient, generalizable, and interpretable deep learning, which we refer to as knowledge-augmented deep learning…
