Federated learning is a machine learning paradigm that emerges as a solution to the privacy-preservation demands in artificial intelligence. As machine learning, federated learning is threatened by adversarial attacks against the integrity of the learning model and the privacy of data via a distributed approach to tackle local and global learning. This weak point is exacerbated by the inaccessibility of data in federated learning, which makes harder the protection against adversarial attacks and evidences the need to furtherance the research on defence methods to make federated learning a real solution for safeguarding data privacy. In this paper, we present an extensive review of the threats of federated learning, as well as as their corresponding countermeasures, attacks versus defences. This survey provides a taxonomy of adversarial attacks and a taxonomy of defence methods that depict a general picture of this vulnerability of federated learning and how to overcome it. Likewise, we expound guidelines for selecting the most adequate defence method according to the category of the adversarial attack. Besides, we carry out an extensive experimental study from which we draw further conclusions about the behaviour of attacks and defences and the guidelines for selecting the most adequate defence method according to the category of the adversarial attack. This study is finished leading to meditated learned lessons and challenges.
Federal Office for Information Security（德国联邦信息安全办公室）
This article deals with the IT security of connectionist artificial intelligence (AI) applications, focusing on threats to integrity, one of the three IT security goals. Such threats are for instance most relevant in prominent AI computer vision applications. In order to present a holistic view on the IT security goal integrity, many additional aspects such as interpretability, robustness and documentation are taken into account. A comprehensive list of threats and possible mitigations is presented by reviewing the state-of-the-art literature. AI-specific vulnerabilities such as adversarial at…
We describe a threat model under which a split network-based federated learning system is susceptible to a model inversion attack by a malicious computational server. We demonstrate that the attack can be successfully performed with limited knowledge of the data distribution by the attacker. We propose a simple additive noise method to defend against model inversion, finding that the method can significantly reduce attack efficacy at an acceptable accuracy trade-off on MNIST. Furthermore, we show that NoPeekNN, an existing defensive method, protects different information from exposure, suggest…
Murdoch University（莫道克大学）；University of Western Australia（西澳大学）
Deep learning has become an integral part of various computer vision systems in recent years due to its outstanding achievements for object recognition, facial recognition, and scene understanding. However, deep neural networks (DNNs) are susceptible to be fooled with nearly high confidence by an adversary. In practice, the vulnerability of deep learning systems against carefully perturbed images, known as adversarial examples, poses a dire security threat in the physical world applications. To address this phenomenon, we present, what to our knowledge, is the first ever image set based advers…
Preventing Adversarial AI Attacks Against Autonomous Situational Awareness: A Maritime Case Study
The School of Engineering, Computing and Mathematics, University of Plymouth（工程、计算与数学学院，普利茅斯大学）；The Alan Turing Institute, British Library（艾伦·图灵研究所，大英图书馆）
FedLAD: A Linear Algebra Based Data Poisoning Defence for Federated Learning
School of Computing Technologies, Centre of Cyber Security Research and Innovation (CCSRI), RMIT University（计算技术学院，网络安全研究与创新中心（CCSRI），皇家墨尔本理工学院）；School of Computing Technologies, Centre of Cyber Security Research（计算技术学院，网络安全研究中心）；Cyber Security discipline, School of Information Technologies, Deakin University（网络安全学科，信息科技学院，迪金大学）
The critical assessment presented within this paper explores existing research pertaining to the Advanced Persistent Threat (APT) branch of cyber security, applying the knowledge extracted from this research to discuss, evaluate and opinionate upon the areas of discussion as well as involving personal experiences and knowledge within this field. The synthesis of current literature delves into detection capabilities and techniques as well as defensive solutions for organisations with respect to APTs. Higher-tier detection and defensive strategies bear greater importance with larger organisation…
Modern electric power grid, known as the Smart Grid, has fast transformed the isolated and centrally controlled power system to a fast and massively connected cyber-physical system that benefits from the revolutions happening in the communications and the fast adoption of Internet of Things devices. While the synergy of a vast number of cyber-physical entities has allowed the Smart Grid to be much more effective and sustainable in meeting the growing global energy challenges, it has also brought with it a large number of vulnerabilities resulting in breaches of data integrity, confidentiality …
Learning to Communicate in Multi-Agent Reinforcement Learning for Autonomous Cyber Defence
Electrical and Computer Engineering Department, Royal Military College of Canada（加拿大皇家军事学院电子与计算机工程系）；Defence Research and Development Canada（加拿大国防研究发展署）
本文提出在Cyber Operations Research Gym中利用可微分智能体间学习算法训练防御智能体，使其在部分可观测网络环境中学习低成本通信与防御战术策略，从而提升自主网络防御的协同决策能力。
Our research addresses the question: What are the conditions of the UK's cyber threat landscape? In addressing this we focus on detectable, known and therefore potentially preventable cyber threats, specifically those that are identifiable by the types of malicious scanning activities they exhibit. We have chosen this approach for two reasons. First, as is evidenced herein, the vast majority of cyber threats affecting the lives and business endeavours of UK citizens are identifiable, preventable threats. Thus the potential exists to better improve UK cyber defence by improving how citizens are…
Machine learning (ML) models are applied in an increasing variety of domains. The availability of large amounts of data and computational resources encourages the development of ever more complex and valuable models. These models are considered intellectual property of the legitimate parties who have trained them, which makes their protection against stealing, illegitimate redistribution, and unauthorized application an urgent need. Digital watermarking presents a strong mechanism for marking model ownership and, thereby, offers protection against those threats. This work presents a taxonomy i…
Social bots represent a new generation of bots that make use of online social networks (OSNs) as a command and control (C\&C) channel. Malicious social bots were responsible for launching large-scale spam campaigns, promoting low-cap stocks, manipulating user's digital influence and conducting political astroturf. This paper presents a detailed review on current social bots and proper techniques that can be used to fly under the radar of OSNs defences to be undetectable for long periods of time. We also suggest a refined taxonomy of detection approaches from social network perspective, as well…
National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）；Sumitomo Electric Industries, Ltd.（住友电气工业株式会社）
In this article, we propose the Artificial Intelligence Security Taxonomy to systematize the knowledge of threats, vulnerabilities, and security controls of machine-learning-based (ML-based) systems. We first classify the damage caused by attacks against ML-based systems, define ML-specific security, and discuss its characteristics. Next, we enumerate all relevant assets and stakeholders and provide a general taxonomy for ML-specific threats. Then, we collect a wide range of security controls against ML-specific threats through an extensive review of recent literature. Finally, we classify the…
Bosch Center for Artificial Intelligence, Robert Bosch GmbH（罗伯特·博世有限公司博世人工智能中心）；University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）
Adversarial patch attacks are an emerging security threat for real world deep learning applications. We present Demasked Smoothing, the first approach (up to our knowledge) to certify the robustness of semantic segmentation models against this threat model. Previous work on certifiably defending against patch attacks has mostly focused on image classification task and often required changes in the model architecture and additional training which is undesirable and computationally expensive. In Demasked Smoothing, any segmentation model can be applied without particular training, fine-tuning, o…
Recently security researchers have started to look into automated generation of attack trees from socio-technical system models. The obvious next step in this trend of automated risk analysis is automating the selection of security controls to treat the detected threats. However, the existing socio-technical models are too abstract to represent all security controls recommended by practitioners and standards. In this paper we propose an attack-defence model, consisting of a set of attack-defence bundles, to be generated and maintained with the socio-technical model. The attack-defence bundles …
Adversarial data poisoning is an effective attack against machine learning and threatens model integrity by introducing poisoned data into the training dataset. So far, it has been studied mostly for classification, even though regression learning is used in many mission critical systems (such as dosage of medication, control of cyber-physical systems and managing power supply). Therefore, in the present research, we aim to evaluate all aspects of data poisoning attacks on regression learning, exceeding previous work both in terms of breadth and depth. We present realistic scenarios in which d…
With the development of information technology, the border of the cyberspace gets much broader, exposing more and more vulnerabilities to attackers. Traditional mitigation-based defence strategies are challenging to cope with the current complicated situation. Security practitioners urgently need better tools to describe and modelling attacks for defence. The provenance graph seems like an ideal method for threat modelling with powerful semantic expression ability and attacks historic correlation ability. In this paper, we firstly introduce the basic concepts about system-level provenance grap…
Abertay University（阿伯泰大学）；Naval Academy Research Institute（海军学院研究院）；Middlesex University（密德萨斯大学）；University of Strathclyde（思克莱德大学）
As the world moves towards being increasingly dependent on computers and automation, building secure applications, systems and networks are some of the main challenges faced in the current decade. The number of threats that individuals and businesses face is rising exponentially due to the increasing complexity of networks and services of modern networks. To alleviate the impact of these threats, researchers have proposed numerous solutions for anomaly detection; however, current tools often fail to adapt to ever-changing architectures, associated threats and zero-day attacks. This manuscript …
Fraunhofer AISEC（弗劳恩霍夫应用集成安全研究所）；Technical University of Munich（慕尼黑工业大学）
Data poisoning is one of the most relevant security threats against machine learning and data-driven technologies. Since many applications rely on untrusted training data, an attacker can easily craft malicious samples and inject them into the training dataset to degrade the performance of machine learning models. As recent work has shown, such Denial-of-Service (DoS) data poisoning attacks are highly effective. To mitigate this threat, we propose a new approach of detecting DoS poisoned instances. In comparison to related work, we deviate from clustering and anomaly detection based approaches…
Indian Institute of Technology, Kharagpur（印度理工学院卡拉格布尔分校）；The Ohio State University（俄亥俄州立大学）；Nanyang Technological University（南洋理工大学）
Deep learning has emerged as a strong and efficient framework that can be applied to a broad spectrum of complex learning problems which were difficult to solve using the traditional machine learning techniques in the past. In the last few years, deep learning has advanced radically in such a way that it can surpass human-level performance on a number of tasks. As a consequence, deep learning is being extensively used in most of the recent day-to-day applications. However, security of deep learning systems are vulnerable to crafted adversarial examples, which may be imperceptible to the human …
Pre-trained language models (PTLMs) have achieved great success and remarkable performance over a wide range of natural language processing (NLP) tasks. However, there are also growing concerns regarding the potential security issues in the adoption of PTLMs. In this survey, we comprehensively systematize recently discovered threats to PTLM systems and applications. We perform our attack characterization from three interesting perspectives. (1) We show threats can occur at different stages of the PTLM pipeline raised by different malicious entities. (2) We identify two types of model transfera…
A Comprehensive Review of One-Pixel Attack: Research Status, Taxonomy, Applications, Regulation Policy and Future Directions
Bangladesh University of Business and Technology（孟加拉国商业与技术大学）；Universiti Sains Malaysia（马来西亚理科大学）；Universität Bremen（不来梅大学）；University of Asia Pacific（亚太大学）
Digital Technology National Research Council Canada（加拿大国家研究委员会数字技术研究所）
APT, known as Advanced Persistent Threat, is a difficult challenge for cyber defence. These threats make many traditional defences ineffective as the vulnerabilities exploited by these threats are insiders who have access to and are within the network. This paper proposes DeepTaskAPT, a heterogeneous task-tree based deep learning method to construct a baseline model based on sequences of tasks using a Long Short-Term Memory (LSTM) neural network that can be applied across different users to identify anomalous behaviour. Rather than applying the model to sequential log entries directly, as most…
Cyber threats are increasing not only in their volume but also in their sophistication and difficulty to detect. Attacks have become a national/global threat as they have targeted private and public, as well as government sectors over the years. This is a growing issue and organisations are taking steps to reduce, detect and prevent threats. To do this they need to use systems that are equipped with the capabilities to do either of those steps and develop them for the type of networks they use, for instance wired or wireless. One of these systems are Intrusion Detection Systems (IDS), which ca…
Internet technology is so pervasive today, for example, from online social networking to online banking, it has made people's lives more comfortable. Due the growth of Internet technology, security threats to systems and networks are relentlessly inventive. One such a serious threat is "phishing", in which, attackers attempt to steal the user's credentials using fake emails or websites or both. It is true that both industry and academia are working hard to develop solutions to combat against phishing threats. It is therefore very important that organisations to pay attention to end-user awaren…
Attacks in cyberspace have got attention due to risk at privacy, breach of trust and financial losses for individuals as well as organizations. In recent years, these attacks have become more complex to analyze technically, as well as to detect and prevent from accessing confidential data. Although there are many methodologies and mechanisms which have been suggested for cyber-attack detection and prevention, but not from the perspective of an attacker. This paper presents the cyber-defence as hindrances, faced by the attacker, by understanding attack thread and defence possibilities with exis…
With the increasing number of network threats it is essential to have a knowledge of existing and new network threats in order to design better intrusion detection systems. In this paper we propose a taxonomy for classifying network attacks in a consistent way, allowing security researchers to focus their efforts on creating accurate intrusion detection systems and targeted datasets.
Machine learning models are shown to face a severe threat from Model Extraction Attacks, where a well-trained private model owned by a service provider can be stolen by an attacker pretending as a client. Unfortunately, prior works focus on the models trained over the Euclidean space, e.g., images and texts, while how to extract a GNN model that contains a graph structure and node features is yet to be explored. In this paper, for the first time, we comprehensively investigate and develop model extraction attacks against GNN models. We first systematically formalise the threat modelling in the…
A thorough and systematic understanding of different elements of cyberattacks is essential for developing the necessary tools to prevent, detect, diagnose, and mitigate cyberattacks in manufacturing systems. In response, researchers have proposed several attack taxonomies as methods for recognizing and categorizing various cyberattack attributes. However, those taxonomies cover selected attack attributes depending on the research focus, sometimes accompanied by inconsistent naming and definitions. These seemingly different taxonomies often overlap and can complement each other to create a comp…
IT security community is recently facing a change of trend from closed to open working groups and from restrictive information to full information disclosure and sharing. One major feature for this trend change is the number of incidents and various Indicators of compromise (IoC) that appear on a daily base, which can only be faced and solved in a collaborative way. Sharing information is key to stay on top of the threats. To cover the needs of having a medium for information sharing, different initiatives were taken such as the Open Source Threat Intelligence and Sharing Platform called MISP.…
