Federated learning systems are increasingly threatened by data poisoning attacks, where malicious clients compromise global models by contributing tampered updates. Existing defenses often rely on impractical assumptions, such as access to a central test dataset, or fail to generalize across diverse attack types, particularly those involving multiple malicious clients working collaboratively. To address this, we propose Federated Noise-Induced Activation Analysis (FedNIA), a novel defense framework to identify and exclude adversarial clients without relying on any central test dataset. FedNIA injects random noise inputs to analyze the layerwise activation patterns in client models leveraging an autoencoder that detects abnormal behaviors indicative of data poisoning. FedNIA can defend against diverse attack types, including sample poisoning, label flipping, and backdoors, even in scenarios with multiple attacking nodes. Experimental results on non-iid federated datasets demonstrate its effectiveness and robustness, underscoring its potential as a foundational approach for enhancing the security of federated learning systems.
Princeton University（普林斯顿大学）；University of Chicago（芝加哥大学）；IBM
Federated learning is inherently vulnerable to model poisoning attacks because its decentralized nature allows attackers to participate with compromised devices. In model poisoning attacks, the attacker reduces the model's performance on targeted sub-tasks (e.g. classifying planes as birds) by uploading "poisoned" updates. In this report we introduce \algoname{}, a novel defense that uses global top-k update sparsification and device-level gradient clipping to mitigate model poisoning attacks. We propose a theoretical framework for analyzing the robustness of defenses against poisoning attacks…
Machine learning (ML) over distributed multi-party data is required for a variety of domains. Existing approaches, such as federated learning, collect the outputs computed by a group of devices at a central aggregator and run iterative algorithms to train a globally shared model. Unfortunately, such approaches are susceptible to a variety of attacks, including model poisoning, which is made substantially worse in the presence of sybils. In this paper we first evaluate the vulnerability of federated learning to sybil-based poisoning attacks. We then describe \emph{FoolsGold}, a novel defense to…
University of Maryland, College Park（马里兰大学帕克分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Machine learning algorithms are vulnerable to data poisoning attacks. Prior taxonomies that focus on specific scenarios, e.g., indiscriminate or targeted, have enabled defenses for the corresponding subset of known attacks. Yet, this introduces an inevitable arms race between adversaries and defenders. In this work, we study the feasibility of an attack-agnostic defense relying on artifacts that are common to all poisoning attacks. Specifically, we focus on a common element between all attacks: they modify gradients computed to train the model. We identify two main artifacts of gradients compu…
CSIRO’s Data61（澳大利亚联邦科学与工业研究组织数据61中心）；University of Melbourne（墨尔本大学）
We use distributionally-robust optimization for machine learning to mitigate the effect of data poisoning attacks. We provide performance guarantees for the trained model on the original data (not including the poison records) by training the model for the worst-case distribution on a neighbourhood around the empirical distribution (extracted from the training dataset corrupted by a poisoning attack) defined using the Wasserstein distance. We relax the distributionally-robust machine learning problem by finding an upper bound for the worst-case fitness based on the empirical sampled-averaged f…
Resilience in Online Federated Learning: Mitigating Model-Poisoning Attacks via Partial Sharing
Norwegian University of Science and Technology（挪威科技大学）；Aalto University（阿尔托大学）；Commonwealth Scientific and Industrial Research Organisation（联邦科学与工业研究组织）；University of Southern Denmark（南丹麦大学）
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, the School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院、教育部航空航天信息安全与可信计算重点实验室）；Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology（华中科技大学网络空间安全学院、湖北省大数据安全工程研究中心）；School of Computer Science, Wuhan University（武汉大学计算机学院）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；VMware, Inc.（VMware公司）
Collaborative learning allows multiple clients to train a joint model without sharing their data with each other. Each client performs training locally and then submits the model updates to a central server for aggregation. Since the server has no visibility into the process of generating the updates, collaborative learning is vulnerable to poisoning attacks where a malicious client can generate a poisoned update to introduce backdoor functionality to the joint model. The existing solutions for detecting poisoned updates, however, fail to defend against the recently proposed attacks, especiall…
Mitigating Malicious Attacks in Federated Learning via Confidence-aware Defense
Queen Mary University of London（伦敦玛丽女王大学）；University of Oxford（牛津大学）；Huazhong University of Science and Technology（华中科技大学）；Shandong University of Technology（山东理工大学）
GShield: Mitigating Poisoning Attacks in Federated Learning
Department of Computer Science and Engineering, JAIN (Deemed-to-be University), Kochi, India（计算机科学与工程系，贾因大学（认定大学），印度科钦）；Department of Science and Technological Innovation, University of Eastern Piedmont, V.le Teresa Michel, 11, Alessandria, Italy（科学技术创新系，东皮埃蒙特大学，意大利艾利斯安迪亚）；Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Italy（电气、计算机与生物医学工程系，皮埃蒙特大学，意大利）；Department of Computer Applications, Cochin University of Science and Technology, India（计算机应用系，科钦科学与技术大学，印度）
Data poisoning is a type of adversarial attack on training data where an attacker manipulates a fraction of data to degrade the performance of machine learning model. Therefore, applications that rely on external data-sources for training data are at a significantly higher risk. There are several known defensive mechanisms that can help in mitigating the threat from such attacks. For example, data sanitization is a popular defensive mechanism wherein the learner rejects those data points that are sufficiently far from the set of training instances. Prior work on data poisoning defense primaril…
Sparsification Under Siege: Dual-Level Defense Against Poisoning in Communication-Efficient Federated Learning
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Cyber Science and Technology at Beihang University（北京航空航天大学网络安全科学与技术学院）；School of Cyberspace Science and Technology, Beijing Jiaotong University（北京交通大学网络空间科学与技术学院）；Beijing Key Laboratory of Security and Privacy in Intelligent Transportation（北京智能交通安全与隐私重点实验室）；Zhongguancun Laboratory（中关村实验室）；School of Computing and Information,University of Pittsburgh（匹兹堡大学计算与信息学院）
FLEDGE: Ledger-based Federated Learning Resilient to Inference and Backdoor Attacks
The University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德谷分校）；Technical University of Darmstadt（达姆施塔特工业大学）；KOBIL GmbH（KOBIL有限公司）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Gradient Purification: Defense Against Poisoning Attack in Decentralized Federated Learning
Center for Data Science, Zhejiang University, Hangzhou 310000, China（数据科学中心，浙江大学，杭州）；College of Computer Science, Zhejiang University, Hangzhou 310000, China（计算机科学学院，浙江大学，杭州）
Analyzing and Mitigating Poisoning Attacks on Flow-Based Intrusion Detection
CortAIx Labs, Thales Deutschland（Thales Deutschland）；Grey Solutions；University of Osnabrück（奥斯纳布吕克大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；Ulm University（乌尔姆大学）
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；The Pennsylvania State University（宾夕法尼亚州立大学）
The black-box nature of deep neural networks (DNNs) facilitates attackers to manipulate the behavior of DNN through data poisoning. Being able to detect and mitigate poisoning attacks, typically categorized into backdoor and adversarial poisoning (AP), is critical in enabling safe adoption of DNNs in many application domains. Although recent works demonstrate encouraging results on detection of certain backdoor attacks, they exhibit inherent limitations which may significantly constrain the applicability. Indeed, no technique exists for detecting AP attacks, which represents a harder challenge…
Federated Learning (FL) allows multiple clients to collaboratively train a Neural Network (NN) model on their private data without revealing the data. Recently, several targeted poisoning attacks against FL have been introduced. These attacks inject a backdoor into the resulting model that allows adversary-controlled inputs to be misclassified. Existing countermeasures against backdoor attacks are inefficient and often merely aim to exclude deviating models from the aggregation. However, this approach also removes benign models of clients with deviating data distributions, causing the aggregat…
School of Computer Engineering and Science（计算机工程与科学学院）；Shanghai University（上海大学）
It has been proved that deep neural networks are facing a new threat called backdoor attacks, where the adversary can inject backdoors into the neural network model through poisoning the training dataset. When the input containing some special pattern called the backdoor trigger, the model with backdoor will carry out malicious task such as misclassification specified by adversaries. In text classification systems, backdoors inserted in the models can cause spam or malicious speech to escape detection. Previous work mainly focused on the defense of backdoor attacks in computer vision, little a…
Many machine learning systems rely on data collected in the wild from untrusted sources, exposing the learning algorithms to data poisoning. Attackers can inject malicious data in the training dataset to subvert the learning process, compromising the performance of the algorithm producing errors in a targeted or an indiscriminate way. Label flipping attacks are a special case of data poisoning, where the attacker can control the labels assigned to a fraction of the training points. Even if the capabilities of the attacker are constrained, these attacks have been shown to be effective to signif…
The backdoor or Trojan attack is a severe threat to deep neural networks (DNNs). Researchers find that DNNs trained on benign data and settings can also learn backdoor behaviors, which is known as the natural backdoor. Existing works on anti-backdoor learning are based on weak observations that the backdoor and benign behaviors can differentiate during training. An adaptive attack with slow poisoning can bypass such defenses. Moreover, these methods cannot defend natural backdoors. We found the fundamental differences between backdoor-related neurons and benign neurons: backdoor-related neuron…
University of Alberta（阿尔伯塔大学）；Illinois Institute of Technology（伊利诺伊理工学院）；Michigan State University（密歇根州立大学）
Recent studies on backdoor attacks in model training have shown that polluting a small portion of training data is sufficient to produce incorrect manipulated predictions on poisoned test-time data while maintaining high clean accuracy in downstream tasks. The stealthiness of backdoor attacks has imposed tremendous defense challenges in today's machine learning paradigm. In this paper, we explore the potential of self-training via additional unlabeled data for mitigating backdoor attacks. We begin by making a pilot study to show that vanilla self-training is not effective in backdoor mitigatio…
A backdoor or Trojan attack is an important type of data poisoning attack against deep neural network (DNN) classifiers, wherein the training dataset is poisoned with a small number of samples that each possess the backdoor pattern (usually a pattern that is either imperceptible or innocuous) and which are mislabeled to the attacker's target class. When trained on a backdoor-poisoned dataset, a DNN behaves normally on most benign test samples but makes incorrect predictions to the target class when the test sample has the backdoor pattern incorporated (i.e., contains a backdoor trigger). Here …
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Trustworthy AI is a critical issue in machine learning where, in addition to training a model that is accurate, one must consider both fair and robust training in the presence of data bias and poisoning. However, the existing model fairness techniques mistakenly view poisoned data as an additional bias to be fixed, resulting in severe performance degradation. To address this problem, we propose FR-Train, which holistically performs fair and robust model training. We provide a mutual information-based interpretation of an existing adversarial training-based fairness-only method, and apply this …
Sharpness-Aware Poisoning: Enhancing Transferability of Injective Attacks on Recommender Systems
Key Laboratory of Knowledge Engineering with Big Data（大数据知识工程重点实验室）；Hefei University of Technology（合肥工业大学）；Intelligent Interconnected Systems Laboratory of Anhui Province（安徽省智能互联系统实验室）；National University of Singapore（新加坡国立大学）
Mitigating Backdoors via Decoy Shortcuts and Knowledge Decoupling
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Securing Autonomous Vehicle Systems via Twin-Aware Federated Reinforcement Learning
North Carolina State University（北卡罗来纳州立大学）；University of Louisville（路易斯维尔大学）；University of Maryland（马里兰大学）；University of North Texas（北德克萨斯大学）；Wayne State University（韦恩州立大学）
