Federated learning (FL) is a collaborative learning paradigm allowing multiple clients to jointly train a model without sharing their training data. However, FL is susceptible to poisoning attacks, in which the adversary injects manipulated model updates into the federated model aggregation process to corrupt or destroy predictions (untargeted poisoning) or implant hidden functionalities (targeted poisoning or backdoors). Existing defenses against poisoning attacks in FL have several limitations, such as relying on specific assumptions about attack types and strategies or data distributions or not sufficiently robust against advanced injection techniques and strategies and simultaneously maintaining the utility of the aggregated model. To address the deficiencies of existing defenses, we take a generic and completely different approach to detect poisoning (targeted and untargeted) attacks. We present FreqFed, a novel aggregation mechanism that transforms the model updates (i.e., weights) into the frequency domain, where we can identify the core frequency components that inherit sufficient information about weights. This allows us to effectively filter out malicious updates during local training on the clients, regardless of attack types, strategies, and clients' data distributions. We extensively evaluate the efficiency and effectiveness of FreqFed in different application domains, including image classification, word prediction, IoT intrusion detection, and speech recognition. We demonstrate that FreqFed can mitigate poisoning attacks effectively with a negligible impact on the utility of the aggregated model.
Princeton University（普林斯顿大学）；University of Chicago（芝加哥大学）；IBM
Federated learning is inherently vulnerable to model poisoning attacks because its decentralized nature allows attackers to participate with compromised devices. In model poisoning attacks, the attacker reduces the model's performance on targeted sub-tasks (e.g. classifying planes as birds) by uploading "poisoned" updates. In this report we introduce \algoname{}, a novel defense that uses global top-k update sparsification and device-level gradient clipping to mitigate model poisoning attacks. We propose a theoretical framework for analyzing the robustness of defenses against poisoning attacks…
University of Maryland, College Park（马里兰大学帕克分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Machine learning algorithms are vulnerable to data poisoning attacks. Prior taxonomies that focus on specific scenarios, e.g., indiscriminate or targeted, have enabled defenses for the corresponding subset of known attacks. Yet, this introduces an inevitable arms race between adversaries and defenders. In this work, we study the feasibility of an attack-agnostic defense relying on artifacts that are common to all poisoning attacks. Specifically, we focus on a common element between all attacks: they modify gradients computed to train the model. We identify two main artifacts of gradients compu…
Machine learning (ML) over distributed multi-party data is required for a variety of domains. Existing approaches, such as federated learning, collect the outputs computed by a group of devices at a central aggregator and run iterative algorithms to train a globally shared model. Unfortunately, such approaches are susceptible to a variety of attacks, including model poisoning, which is made substantially worse in the presence of sybils. In this paper we first evaluate the vulnerability of federated learning to sybil-based poisoning attacks. We then describe \emph{FoolsGold}, a novel defense to…
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, the School of Cyber Science and Engineering, Wuhan University（武汉大学网络空间安全学院、教育部航空航天信息安全与可信计算重点实验室）；Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science and Technology（华中科技大学网络空间安全学院、湖北省大数据安全工程研究中心）；School of Computer Science, Wuhan University（武汉大学计算机学院）；State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）；VMware, Inc.（VMware公司）
Collaborative learning allows multiple clients to train a joint model without sharing their data with each other. Each client performs training locally and then submits the model updates to a central server for aggregation. Since the server has no visibility into the process of generating the updates, collaborative learning is vulnerable to poisoning attacks where a malicious client can generate a poisoned update to introduce backdoor functionality to the joint model. The existing solutions for detecting poisoned updates, however, fail to defend against the recently proposed attacks, especiall…
CSIRO’s Data61（澳大利亚联邦科学与工业研究组织数据61中心）；University of Melbourne（墨尔本大学）
We use distributionally-robust optimization for machine learning to mitigate the effect of data poisoning attacks. We provide performance guarantees for the trained model on the original data (not including the poison records) by training the model for the worst-case distribution on a neighbourhood around the empirical distribution (extracted from the training dataset corrupted by a poisoning attack) defined using the Wasserstein distance. We relax the distributionally-robust machine learning problem by finding an upper bound for the worst-case fitness based on the empirical sampled-averaged f…
GShield: Mitigating Poisoning Attacks in Federated Learning
Department of Computer Science and Engineering, JAIN (Deemed-to-be University), Kochi, India（计算机科学与工程系，贾因大学（认定大学），印度科钦）；Department of Science and Technological Innovation, University of Eastern Piedmont, V.le Teresa Michel, 11, Alessandria, Italy（科学技术创新系，东皮埃蒙特大学，意大利艾利斯安迪亚）；Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Italy（电气、计算机与生物医学工程系，皮埃蒙特大学，意大利）；Department of Computer Applications, Cochin University of Science and Technology, India（计算机应用系，科钦科学与技术大学，印度）
Resilience in Online Federated Learning: Mitigating Model-Poisoning Attacks via Partial Sharing
Norwegian University of Science and Technology（挪威科技大学）；Aalto University（阿尔托大学）；Commonwealth Scientific and Industrial Research Organisation（联邦科学与工业研究组织）；University of Southern Denmark（南丹麦大学）
Mitigating Malicious Attacks in Federated Learning via Confidence-aware Defense
Queen Mary University of London（伦敦玛丽女王大学）；University of Oxford（牛津大学）；Huazhong University of Science and Technology（华中科技大学）；Shandong University of Technology（山东理工大学）
Sparsification Under Siege: Dual-Level Defense Against Poisoning in Communication-Efficient Federated Learning
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Cyber Science and Technology at Beihang University（北京航空航天大学网络安全科学与技术学院）；School of Cyberspace Science and Technology, Beijing Jiaotong University（北京交通大学网络空间科学与技术学院）；Beijing Key Laboratory of Security and Privacy in Intelligent Transportation（北京智能交通安全与隐私重点实验室）；Zhongguancun Laboratory（中关村实验室）；School of Computing and Information,University of Pittsburgh（匹兹堡大学计算与信息学院）
Data poisoning is a type of adversarial attack on training data where an attacker manipulates a fraction of data to degrade the performance of machine learning model. Therefore, applications that rely on external data-sources for training data are at a significantly higher risk. There are several known defensive mechanisms that can help in mitigating the threat from such attacks. For example, data sanitization is a popular defensive mechanism wherein the learner rejects those data points that are sufficiently far from the set of training instances. Prior work on data poisoning defense primaril…
FLEDGE: Ledger-based Federated Learning Resilient to Inference and Backdoor Attacks
The University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德谷分校）；Technical University of Darmstadt（达姆施塔特工业大学）；KOBIL GmbH（KOBIL有限公司）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Analyzing and Mitigating Poisoning Attacks on Flow-Based Intrusion Detection
CortAIx Labs, Thales Deutschland（Thales Deutschland）；Grey Solutions；University of Osnabrück（奥斯纳布吕克大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；Ulm University（乌尔姆大学）
Gradient Purification: Defense Against Poisoning Attack in Decentralized Federated Learning
Center for Data Science, Zhejiang University, Hangzhou 310000, China（数据科学中心，浙江大学，杭州）；College of Computer Science, Zhejiang University, Hangzhou 310000, China（计算机科学学院，浙江大学，杭州）
Federated Learning (FL) allows multiple clients to collaboratively train a Neural Network (NN) model on their private data without revealing the data. Recently, several targeted poisoning attacks against FL have been introduced. These attacks inject a backdoor into the resulting model that allows adversary-controlled inputs to be misclassified. Existing countermeasures against backdoor attacks are inefficient and often merely aim to exclude deviating models from the aggregation. However, this approach also removes benign models of clients with deviating data distributions, causing the aggregat…
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；The Pennsylvania State University（宾夕法尼亚州立大学）
The black-box nature of deep neural networks (DNNs) facilitates attackers to manipulate the behavior of DNN through data poisoning. Being able to detect and mitigate poisoning attacks, typically categorized into backdoor and adversarial poisoning (AP), is critical in enabling safe adoption of DNNs in many application domains. Although recent works demonstrate encouraging results on detection of certain backdoor attacks, they exhibit inherent limitations which may significantly constrain the applicability. Indeed, no technique exists for detecting AP attacks, which represents a harder challenge…
School of Computer Engineering and Science（计算机工程与科学学院）；Shanghai University（上海大学）
It has been proved that deep neural networks are facing a new threat called backdoor attacks, where the adversary can inject backdoors into the neural network model through poisoning the training dataset. When the input containing some special pattern called the backdoor trigger, the model with backdoor will carry out malicious task such as misclassification specified by adversaries. In text classification systems, backdoors inserted in the models can cause spam or malicious speech to escape detection. Previous work mainly focused on the defense of backdoor attacks in computer vision, little a…
Many machine learning systems rely on data collected in the wild from untrusted sources, exposing the learning algorithms to data poisoning. Attackers can inject malicious data in the training dataset to subvert the learning process, compromising the performance of the algorithm producing errors in a targeted or an indiscriminate way. Label flipping attacks are a special case of data poisoning, where the attacker can control the labels assigned to a fraction of the training points. Even if the capabilities of the attacker are constrained, these attacks have been shown to be effective to signif…
The backdoor or Trojan attack is a severe threat to deep neural networks (DNNs). Researchers find that DNNs trained on benign data and settings can also learn backdoor behaviors, which is known as the natural backdoor. Existing works on anti-backdoor learning are based on weak observations that the backdoor and benign behaviors can differentiate during training. An adaptive attack with slow poisoning can bypass such defenses. Moreover, these methods cannot defend natural backdoors. We found the fundamental differences between backdoor-related neurons and benign neurons: backdoor-related neuron…
A backdoor or Trojan attack is an important type of data poisoning attack against deep neural network (DNN) classifiers, wherein the training dataset is poisoned with a small number of samples that each possess the backdoor pattern (usually a pattern that is either imperceptible or innocuous) and which are mislabeled to the attacker's target class. When trained on a backdoor-poisoned dataset, a DNN behaves normally on most benign test samples but makes incorrect predictions to the target class when the test sample has the backdoor pattern incorporated (i.e., contains a backdoor trigger). Here …
Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuning
Nanyang Technological University（南洋理工大学）；Hong Kong University of Science and Technology（香港科技大学）；Jinan University（暨南大学）；Sony Research（索尼研究院）；Zhejiang University（浙江大学）；Beijing Institute of Technology（北京理工大学）
University of Alberta（阿尔伯塔大学）；Illinois Institute of Technology（伊利诺伊理工学院）；Michigan State University（密歇根州立大学）
Recent studies on backdoor attacks in model training have shown that polluting a small portion of training data is sufficient to produce incorrect manipulated predictions on poisoned test-time data while maintaining high clean accuracy in downstream tasks. The stealthiness of backdoor attacks has imposed tremendous defense challenges in today's machine learning paradigm. In this paper, we explore the potential of self-training via additional unlabeled data for mitigating backdoor attacks. We begin by making a pilot study to show that vanilla self-training is not effective in backdoor mitigatio…
In federated learning, machine learning and deep learning models are trained globally on distributed devices. The state-of-the-art privacy-preserving technique in the context of federated learning is user-level differential privacy. However, such a mechanism is vulnerable to some specific model poisoning attacks such as Sybil attacks. A malicious adversary could create multiple fake clients or collude compromised devices in Sybil attacks to mount direct model updates manipulation. Recent works on novel defense against model poisoning attacks are difficult to detect Sybil attacks when different…
Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation
The City University of New York, CSI, Staten Island, NY 10314, USA（纽约城市大学，CSI，史泰登岛分校）；The City University of New York, Graduate Center, New York, NY 10016, USA（纽约城市大学研究生中心）
ETDI: Mitigating Tool Squatting and Rug Pull Attacks in Model Context Protocol (MCP) by using OAuth-Enhanced Tool Definitions and Policy-Based Access Control
Amazon Independent Researcher（亚马逊独立研究者）；Project Kuiper Security (KPES)（项目Kuiper安全）；Proactive Security OWASP, Amazon Web Services（主动安全OWASP，亚马逊网络服务）；Intuit Adversarial AI Security reSearch (A2RS)（Intuit对抗AI安全reSearch）
本文提出 ETDI 安全扩展，通过密码学身份验证、不可变版本化工具定义、OAuth 2.0 权限管理及基于策略的细粒度访问控制，缓解 MCP 中的工具投毒和 Rug Pull 攻击，构建更安全可控的 AI 工具交互生态。
